Locating objects in world coordinate system

ABSTRACT

A technique for detecting a velocity state of a device includes generating multiple phase measurements for each of multiple packets emitted by the device and monitoring differences between phase measurements made for different packets. The technique further includes asserting a particular velocity state of the device based on a condition of the monitored differences.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/422,073, filed May 24, 2019, which claims the benefit of U.S.Provisional Application No. 62/684,586, filed Jun. 13, 2018, and of U.S.Provisional Application No. 62/803,075, filed Feb. 8, 2019. The contentsand teachings of each of the above applications are incorporated hereinby reference in their entirety.

BACKGROUND

Augmented Reality (AR) systems are beginning to ship in volume andinclude mobile phone-based systems such as Apple's ARKit and Google'sARCore, as well as head mounted displays (HMDs) such as the MicrosoftHololens. Existing AR systems use computer vision to create a spatialunderstanding map of the user's environment and create spatial anchorsfor hologram placement. Each spatial anchor is the origin of acoordinate system, and its position is adjusted over time relative toother spatial anchors in the user's environment. The purpose of spatialanchors is to anchor holograms in 3D space and re-localize the hologramto adjacent spatial anchors if the user moves the hologram throughspace. Each spatial anchor is also a point in every other spatialanchor's coordinate system, thus establishing what can be considered anoverall coordinate system.

SUMMARY

Unfortunately, spatial anchors involve certain deficiencies. Forexample, the relationship between spatial anchors is imprecise. Also,the relationship between spatial anchors shifts continually with time.What is needed is a more precise and stable solution.

In contrast with prior approaches, certain embodiments are directed to atechnique that includes constructing a WCS (world coordinate system) asa three-dimensional, Cartesian coordinate system based at least in parton (i) a plurality of anchor points detected in a physical space arounda device and (ii) a secondary source of information that assists inlocalizing objects in the physical space. The technique further includesproviding the WCS to the device as a coordinate system in which to placeholograms.

Other embodiments are directed to a technique of locating objects inthree-dimensional space, including constructing a WCS (world coordinatesystem) based on a plurality of VISOs (visually-identifiable stationaryobjects) disposed in respective locations of a physical space, andproviding the WCS to a device as a coordinate system in which to placeholograms.

The above-described techniques may be embodied as methods. They may alsobe embodied as one or more computerized systems constructed and arrangedto perform any of the above-described techniques. Such techniques mayfurther be embodied as a computer program product. The computer programproduct stores instructions which, when executed on control circuitry ofa computerized apparatus, cause the computerized apparatus to performany of the above-described methods.

The foregoing summary is presented for illustrative purposes to assistthe reader in readily grasping example features presented herein;however, this summary is not intended to set forth required elements orto limit embodiments hereof in any way. One should appreciate that theabove-described features can be combined in any manner that makestechnological sense, and that all such combinations are intended to bedisclosed herein, regardless of whether such combinations are identifiedexplicitly or not.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The foregoing and other features and advantages will be apparent fromthe following description of particular embodiments, as illustrated inthe accompanying drawings, in which like reference characters refer tothe same or similar parts throughout the different views. The drawingsare not necessarily to scale, emphasis instead being placed uponillustrating the principles of various embodiments.

FIG. 1 is a block diagram of an example environment in which embodimentsof the improved technique can be practiced.

FIG. 2 is a block diagram of an example server apparatus used in theenvironment of FIG. 1.

FIG. 3 is a block diagram of an example head-mounted display (HMD) usedin the environment of FIG. 1.

FIG. 4 is a block diagram of an example device (e.g., wand) used in theenvironment of FIG. 1.

FIG. 5 is a flowchart showing an example method of generating an offsetthat represents a difference between a spatial direction of a device anda local magnetic field direction as reported by the device.

FIG. 6 is a flowchart showing an example method of applying an offset,such as the offset generated in FIG. 5, to correct a pointing directionof a device given the direction of the local magnetic field as reportedby the device.

FIG. 7 is a block diagram of the example environment of FIG. 1, furthershowing a sharing of offsets among devices and/or users.

FIG. 8 is a flowchart showing an example method of measuring a yawdirection of a device in a space.

FIG. 9 is a block diagram of the example environment of FIG. 1, furthershowing a detection of an at-rest state of a device based at least inpart on signals emitted by the device.

FIG. 10 is a perspective view of an example antenna array of FIG. 9.

FIG. 11 is a graph of unwrapped CSI (Channel State Information) phase,as derived from any of the antennas in the antenna array of FIG. 10.

FIGS. 12a and 12b respectively show another perspective view of theantenna array of FIG. 10 and a flowchart showing a method of processingCSI phase to detect an at-rest condition of a device.

FIG. 13 is a block diagram of an example arrangement for determiningwhen a dead-stop state is both entered and exited.

FIG. 14 is a top view of a device that is adapted for accuratemeasurements of orientation and movement.

FIG. 15 is a flowchart showing an example method of detecting a velocitystate of a device.

FIG. 16 is a diagram showing an example of drift that a deviceexperiences in movement from between first and second WAPs.

FIG. 17 is a diagram showing an example of accumulate drift.

FIG. 18 is a diagram showing an example device drift correction method.

FIG. 19 is a diagram showing example anchor point position andorientation errors.

FIG. 20 is a diagram showing an example calculation of an alpha value ofan automatically rebuilt anchor point.

FIG. 21 is a diagram showing an example magnitude of an error caused bya lever arm effect at an i'th existing anchor point.

FIG. 22 is a block diagram showing multiple example devices sharing acommon language.

FIG. 23 is a block diagram showing example WCS persistent anchor pointsvs. non-WCS temporal anchor points.

FIG. 24 is a diagram showing a perceived similarity between a smallclose sphere and a big distant sphere.

DETAILED DESCRIPTION

Embodiments of the improved technique will now be described. One shouldappreciate that such embodiments are provided by way of example toillustrate certain features and principles but are not intended to belimiting.

This disclosure is presented in the following sections:

-   -   Section I describes example techniques for measuring a yaw        direction of a device in space;    -   Section II describes example techniques for detecting a velocity        state of a device; and    -   Section III describes example techniques for locating objects in        a World Coordinate System (WCS).

Section I: Yaw Measurement

This section describes an improved technique for measuring yaw(left-right direction) of a device. The technique includes obtaining afirst measurement of an orientation of the device relative to a localmagnetic frame of reference (FoR) and a second measurement of theorientation of the device relative to a spatial FoR, with bothmeasurements made while the device is disposed at a known location andin the same orientation. The technique further includes computing anoffset between the two measurements and storing the offset in connectionwith the known location. When the device later returns to the samelocation, the yaw direction of the device is determined based onacquiring a new measurement of the device's orientation in the localmagnetic FoR and applying the offset as a correction.

FIG. 1 shows an example environment 100 in which embodiments of theimproved technique can be practiced. As shown, the environment 100includes a physical space 102, such as a room, building, or other indooror outdoor area. The space 102 is permeated by a magnetic field 104,which tends to vary over space, as shown by the differently-pointingarrows. Although depicted for neatness over only a small patch of thespace 102, the magnetic field 104 extends throughout the entire space102 and roughly aligns with the Earth's magnetic field 106. Variabilityin the magnetic field 104 as measured in the space 102 may arise fromnearby objects that have magnetic properties. To the extent that theseobjects do not change location or operating state over time (anappliance might produce different magnetic effects when on than whenoff), the magnetic field 104 remains stationary, varying over space butnot significantly over time. The magnetic field 104 in the space 102 canthus be regarded as quasi-stationary.

As further shown, a user 110 in the space 102 wears a head-mounteddisplay (HMD) 120 and holds a device 130. The HMD 120 may be VR (VirtualReality) goggles, AR (Augmented Reality) goggles, MR (Mixed Reality)goggles, or the like, with suitable examples including the MicrosoftHoloLens, Oculus Rift, or Magic Leap. The HMD 120 may include a camera,such as a wide-field depth camera 122, which the HMD 120 uses forimaging its surroundings in three dimensions. For example, the depthcamera 122 generates images and projects pulses of light in theenvironment 100. The depth camera 122 uses reflections of those pulsesto detect depth, such that pixels in the generated images haveassociated depth values. The field of view of the depth camera 122 istypically wide enough to include the device 130, even if the user 110 isnot looking directly at the device 130 through the HMD 120. Embodimentsare not limited to depth cameras, to depth cameras that operate asdescribed, or even to cameras, however.

Both the HMD 120 and the device 130 are capable of wirelesscommunication, such as using Wi-Fi and/or Bluetooth. The device 130includes an IMU (Inertial Measurement Unit) configured to measure thedevice's orientation in space, e.g., using a gravity sensor and amagnetometer. The magnetometer is configured to sense the local magneticfield 104 and to establish a local magnetic frame of reference (FoR),which enables the device 130 to orient itself in a yaw (left-right)direction 134, such as by measuring its own yaw angle with respect tothe direction of local magnetic north. In a particular example, device130 is a wireless pointing device configured to project a virtual ray132 for selecting and controlling objects, such as hologram 150 (avirtual object) or TV 152 (a physical object). Ideally, the virtual ray132 aligns with a long axis 138 of the device 130. The pointing device130 may construct the virtual ray 132 based on knowledge of its ownlocation and knowledge of the angle in which it is pointing. An exampleof a pointer device that meets this description is disclosed in U.S.patent application Ser. No. 15/655,489, filed Jul. 20, 2017, thecontents and teachings of which are incorporated herein by reference intheir entirety. Embodiments are not limited to this type of pointingdevice, however, but rather may include any type of device capable ofsensing and reporting its own yaw direction relative to a local magneticfield. Many common devices satisfy these requirements, such as certainsmart phones, tablet computers, PDAs (Personal Data Assistants), gamingconsoles, remote controls, and the like.

Disposed at one or more locations around the space 102 are antennaarrays 160. Three antenna arrays 160 a, 160 b, and 160 c are shown, butthe space 102 may include any number of antenna arrays 160, includingonly a single antenna array. Each antenna array 160 includes one or moreantennas and connects to a server apparatus, or simply “server,” 170.Details of the antenna arrays 160 are not critical for purposes of yawcorrection, as any conventional Wi-Fi (IEEE 802.11) or Bluetooth antennawill suffice. In some examples, an antenna array 160 is integrated withthe server 170 in a single assembly. As shown, server 170 includes a yawprocessor 172 and a data structure 180.

In example operation, the user 110 moves around the space 102, wearingthe HMD 120 and holding the device 130. The user 110 may wish to use thedevice 130 as a pointer to various objects, such as hologram 150 or TV152 (the HMD 120 may render the hologram 150 to the user). A problemarises, however, in that the device 130 on its own can only determineits yaw direction 134 relative to its own local magnetic north, e.g.,based on magnetic north as measured by the magnetometer in its IMU. Themagnetometer in device 130 may be accurate for measuring local magneticnorth, but that local magnetic north may not be pointing to truemagnetic north and typically varies from one place to another within thespace 102. Given that the pointing accuracy of the virtual ray 132 isonly as good as the knowledge of yaw direction 134, erroneous knowledgearising from variations in magnetic field 104 can cause large pointingerrors.

To address these errors, embodiments map the local magnetic field 104over portions of the space 102. Mapping may be accomplished by obtainingmeasurements of the device's orientation in a local magnetic FoR 108 aand comparing them with measurements, made by a separate instrument, ofthe device's orientation in a spatial FoR 108 b, thus producingrespective yaw offsets 190. In an example, the local magnetic FoR 108 ais based on a magnetic sensor and a gravity sensor in the device 130,and the spatial FoR 108 b is based on geometry of the space 102. SpatialFoR 108 b is expressed in a coordinate system of the space 102, which wecall a World Coordinate System, or “WCS.” The WCS may be an X-Y-Zcoordinate system, or some other spatial coordinate system. We refer toan orientation measurement made in the local magnetic FoR 108 a as a“local magnetic yaw” 140 a, and to an orientation measurement in thespatial FoR 108 b as a “reference yaw” 140 b. Each yaw offset 190 canthus be regarded as a difference between a reference yaw 140 b and alocal magnetic yaw 140 a. The server 170 stores yaw offsets 190 in datastructure 180 in connection with corresponding locations 192. Later,when the device 130 returns to the same locations, the server 170 mayapply the yaw offsets 190 to correct new measurements of local magneticyaw 140 a at the same locations, without the need to obtain newmeasurements of reference yaw 140 b. Over time, the data structure 180realizes an offset map that enables devices to estimate their true yawdirections merely by measuring their own local magnetic yaw 140 a andapplying offsets 190 for the respective locations as corrections.

Reference yaw values 140 b may be provided by any observer that iscapable of imaging the device 130 and expressing its spatial yaw in thespatial (WCS) FoR 108 b. We have found that the depth camera 122 iswell-suited for this task, as the HMD 120, which contains the depthcamera 122, generally has possession of its own location and orientationin the spatial FoR 108 b. By imaging the device 130 using the depthcamera 122, the HMD 120 can process the image and calculate values ofreference yaw 140 b of the device 130 relative to the spatial FoR 108 b.

Certain optimizations may facilitate this task. For example, the device130 may be equipped with markings 136, such as shiny black regions,which the depth camera 122 interprets as distant areas or holes. Thedepth holes left by the markings 136 enable the HMD 120 to calculate thelocation and orientation of the device 130 relative to a local FoR ofthe HMD 120, and to translate that location into the spatial FoR 108 b.One should appreciate that embodiments are not limited to devices havingmarkings or to the use of depth cameras. For example, other cameras orimaging instruments having known locations may be used, and thoseinstruments may rely on features other than markings 136 for locatingand orienting the device 130.

As the user 110 moves to different locations in the space 102, themagnetic sensor in the device 130 measures yaw directions of the device130 and the device 130 processes those yaw directions to produce valuesof local magnetic yaw 140 a. At the same locations, the HMD 120 capturesrespective images of the device 130 using the depth camera 122 andgenerates respective values of reference yaw 140 b. The two measurements140 a and 140 b for a given location preferably derive from an image anda magnetic sample acquired simultaneously, or nearly so, so that eachmeasurement pair {140 a, 140 b} reflects the same location 192 and thesame physical orientation of the device 130. In an example, device 130sends its measurements of magnetic yaw 140 a to the server 170, and HMD120 does likewise for its measurements of reference yaw 140 b. For eachpair of measurements {140 a, 140 b}, the server 170 computes arespective yaw offset 190 (e.g., as 140 b minus 140 a) and stores theyaw offset 190 in connection with the respective location 192, i.e., thelocation of the device 130 at which the image and magnetic sample forthat measurement pair were acquired.

Over time, the server 170 fills the data structure 180 with offsetvalues 190 for the respective locations 192. As individual offset valuesare likely to have significant noise, the server 170 may apply averagingto promote smoothness. For example, if a location 192 is visited morethan once, the server 170 may generate a new offset 190 for thatlocation and average the new offset with the existing one. In someexamples, the data structure 180 maintains a count of the number ofvisits to each location 192 and computes the average as a weightedaverage. For instance, if the current visit to a location is the tenthvisit to that location, the new offset may be given a weight ofone-tenth. In some examples, old values of offset are given lowerweights than newer ones, allowing the data structure 180 to adapt tochanges in local magnetic field 104 by aging out older values. In someexamples, the data structure 180 also stores a timestamp that indicatesthe last time a new offset at each location 192 was generated. Very oldentries, as indicated by timestamp, may be aged out more aggressivelythan newer ones.

In some examples, particularly those in which pointing accuracy isespecially critical, the server 170 waits for some level of offsetaveraging to occur before it allows offsets to be used for yawcorrection. For example, the server 170 may wait for the count at acurrent location to exceed a threshold, may wait for a certain amount oftime to pass (as measured by a difference between the current time andthe timestamp for that location), and/or may wait until the offset atthe current location changes by more than a specified amount.

In some examples, the server 170 also uses averaging when responding tooffset requests. For example, when the device 130 enters apreviously-visited location, the device can measure its local magneticyaw 140 a and contact the server 170 for the corresponding offset forthat location. Rather than responding with only that one offset value(the one stored for the current location) the server 170 may insteadcompute a spatial average of the current offset with offsets of itsneighbors, where a “neighbor” is a location adjacent to the currentlocation. Depending on the dimensional granularity of the data structure180, only immediately adjacent neighbors may be averaged together, orneighbors within a specified bounding region may be averaged together.The averaging need not be uniform. For example, closer offsets may begiven higher weights in the averaging than more distant ones, andoffsets with higher counts and/or more recent timestamps may be givenhigher weights than those with lower counts and/or older timestamps,which are less likely to be reliable.

Sometimes, the device 130 enters a location 192 for which no offset 190has been stored, but the depth camera 122 or other instrument is blocked(or otherwise unavailable) and cannot image the device 130. In suchcases, the server 170 may estimate an offset for the current locationbased on offsets of the current location's neighbors. For example, theserver 170 may average neighboring offsets, giving higher weights tocloser, newer, and/or more-often visited locations than to those thatare less so. Thus, the server 170 is capable of producing a correctedyaw measurement for a location, even in the absence of any reference yaw140 b of the device 130 at that location.

Over time, acquiring measurements of reference yaw 140 b (e.g., fromdepth camera 122) may become less necessary, as offsets in the datastructure 180 tend to stabilize. Then, the device 130 may rely onoffsets 190 as being correct and simply apply the offsets to its localmagnetic yaw measurements 140 a to generate corrected values of yaw,checking them with new reference yaw measurements 140 b onlyoccasionally. In some examples, the server 170 may perform spatialaveraging of offsets as a general rule when responding to offsetrequests. Such spatial averaging has the effect of smoothing pointerdirection when the device 130 is moving and helps to prevent suddenjumps. In some examples, the particulars of spatial averaging depend ondetected motion. For example, if it is known that the device 130 isstationary (e.g., using the techniques described in Section II), thespatial averaging may be uniform in direction and may cover only a smallregion surrounding the current location. However, if it is known thatthe device 130 is moving in a particular direction, then spatialaveraging may be biased in favor of the known direction, giving moreweight to offsets at locations in the direction of motion than tooffsets in other directions.

Although the server 170 may generate each value of offset 190 based on asingle image and a single magnetic sample, some embodiments use multipleimages and/or magnetic samples to generate a single offset value. Forexample, the server 170 may provide one or more Kalman filters toestimate and smooth measurements of local magnetic yaw 140 a and/orreference yaw 140 b. Also, if the device 130 is known to be stationaryat a particular moment, the server 170 may leverage this knowledge ofthe stationary state to narrow the variance of the Kalman filter(s)and/or to perform other averaging, filtering, and/or processing, forimproving the accuracy of the offset value by reducing its noise.

FIGS. 2-4 show more detailed views of the server 170, HMD 120, anddevice 130, respectively. As shown in FIG. 2, the server 170 includesone or more communication interfaces 210, such as Wi-Fi and/or Bluetoothinterfaces and an Ethernet interface, a user interface 220 (e.g., mouse,keyboard, monitor, etc.), a set of processors 230 (e.g., one or moreprocessor chips, co-processors, and/or assemblies), and memory 240. Thememory 240 may include both volatile memory, such as random-accessmemory (RAM), and non-volatile memory, such as one or more disk drives,solid state drives, or the like. The set of processors 230 and thememory 240 form control circuitry, which is constructed and arranged tocarry out various methods and functions as described herein. Also, thememory 240 includes a variety of software constructs realized in theform of executable instructions. When the executable instructions arerun by the set of processors 230, the set of processors 230 carry outthe operations of the software constructs. Although certain softwareconstructs are specifically shown and described herein, it is understoodthat the memory 240 typically includes many other software constructs,which are not shown, such as an operating system, various applications,processes, and daemons.

As further shown in FIG. 2, the memory 240 “includes,” i.e., realizes byexecution of software instructions, the above-mentioned yaw processor172 and data structure 180, as well as extended Kalman filters 250 and azero-velocity processor (ZVP) 260.

The extended Kalman filters (EKFs) 250 are configured to estimate andsmooth measures of device orientation and motion in the presence ofnoisy inputs. In this example, extended Kalman filters are preferredover conventional ones, as extended Kalman filters are better athandling non-linearity, which is common in cases of rotational movement.In the example shown, the EKFs include an orientation-only EKF (OOEKF)250 a, a tracking EKF (TEKF) 250 b, and a velocity EKF (VEKF) 250 c. TheOOEKF 250 a is configured to receive values of local magnetic yaw 140 aand to track orientation of the device 130 in its local magnetic FoR 108a, preferably tracking no other characteristics of the device 130, suchas its position or velocity. Limiting the application of the OOEKF 250 ato orientation-only promotes stable tracking of device orientation inthe local magnetic FoR 108 a. In addition to providing a filteredversion of local magnetic yaw 140 a, for purposes of yaw correction, theOOEKF 250 a also provides input to the ZVP 260, e.g., to help determinea velocity state of the device 130.

The TEKF 250 b is configured to receive values of reference yaw 140 b,as well as IMU input, and to track both orientation and location of thedevice 130 in the spatial FoR 108 b. The TEKF 250 b provides a processedversion of reference yaw values 140 b for performing yaw correction. Italso provides estimates of full device orientation (e.g., yaw, pitch,and roll) and device locations 192, which inform the server 170 as tothe locations 192 at which offsets 190 are to be stored and/orretrieved.

The VEKF 250 c is configured to track the full pose (position andorientation) of the device 130 in the local magnetic FoR 108 a. In anexample, the VEKF 250 c performs no direct role in yaw measurement butis rather relevant to determination of velocity state, which isdescribed more fully in Section II.

Shown to the right of FIG. 2 are further example details of the datastructure 180. Here, data structure 180 may be arranged as a spatialdata structure, e.g., as one that provides a respective index for eachspatial dimension, such as X, Y, and Z, which may correspond to thedimensions of the WCS. Arranging the data structure 180 in this mannerpromotes fast lookups and provides a simple basis for locatingneighbors, e.g., for purposes of averaging offset values.

As shown at the bottom-right of FIG. 2, each element of the datastructure 180 may include the computed offset 190 for the respectivelocation, as well as a count 190 a and a timestamp 190 b. In an example,the count 190 a stores the number of offset values that have beenaveraged together to produce the respective offset 190. The timestamp190 b reflects the time of last update of the respective offset 190.

The data structure 180 may represent the space 102 at any desired levelof granularity, such as 10-cm cubes, 1-cm cubes, and so forth, limitedonly by the stability of the WCS. In some examples, the data structure180 is arranged hierarchically, with cubes representing regions and eachregion including multiple elements. Many variations are contemplated.

FIG. 3 shows certain features of the HMD 120 in additional detail. Asshown, HMD 120 includes the above-described depth camera 122, as well asa wireless interface 310 (e.g., Wi-Fi and/or Bluetooth), an IMU 320, aset of processors 330, and memory 340. The memory 340 may include a SLAM(Stabilization, Localization, and Mapping) system 350 and a referenceyaw generator 360. The SLAM system 350 is configured to locate andorient the HMD 120 in space, e.g., based on inputs from the depth camera122 and IMU 320. The reference yaw generator 360 is configured toanalyze images of the device 130, as acquired by the depth camera 122,to apply computer-vision techniques to determine the yaw direction ofthe device 130 relative to a local FoR of the HMD 120, and to transformthat yaw direction into values of reference yaw 140 b relative to thespatial FoR 108 b. The processor(s) 330 and memory 340 form controlcircuitry, which is constructed and arranged to carry out the functionsof the SLAM system 350 and reference yaw generator 360.

FIG. 4 shows certain features of the device 130 in additional detail. Asshown, device 130 includes a wireless interface 410 (e.g., Wi-Fi and/orBluetooth), an IMU 420, a set of processors 430, and memory 440. The IMU420 includes a magnetic sensor 422, such as a magnetometer, and thememory 440 includes a local yaw generator 450. The local yaw generator450 is configured to receive input from the magnetic sensor 422 and toproduce values of local magnetic yaw 140 a. The processor(s) 430 andmemory 440 form control circuitry, which is constructed and arranged tocarry out the functions of the local yaw generator 450. The device 130may also include markings 136, as were introduced in FIG. 1.

FIG. 5 shows an example method 500 for generating yaw offset values andinvolves activities performed by the device 130, the HMD 120, and theserver 170. Although the acts of method 500 are shown in a particularorder, the order shown is merely an example, as the method 500 may beperformed in orders different from that shown, which may includeperforming some acts simultaneously.

At 510, the device 130 (e.g., the “wand”) obtains its yaw directionrelative to local magnetic north as detected by the magnetic sensor 422.At 520, the local yaw generator 450 in the device 130 processes theinput from the magnetic sensor 422 and generates a measure 140 a oflocal magnetic yaw, which it reports to the server 170. The measure 140a of local magnetic yaw is referenced to the local magnetic FoR 108 a,which is based on the device's own measure of magnetic north and on itsobserved direction of gravity, e.g., as read by its IMU 420.

At or about the same time that the device 130 is performing thesefunctions, the HMD 120 performs corresponding functions 530 and 540. At530, the HMD 120 images the device 130 using the depth camera 122. At540, the reference yaw generator 360 computes the reference yaw 140 b ofthe device 130 in the spatial FoR 108 b.

At 560, the server 170 receives the local magnetic yaw 140 a andprocesses the local magnetic yaw 140 a using the OOEKF 250 a. Likewise,at 570 the server 170 receives the reference yaw 140 b and processes thereference yaw 140 b using the TEKF 250 b. At 580, the server 170computes the offset 190, e.g., as the difference between the processedversions of the reference yaw 140 b and the local yaw 140 a.

At 590, the server 170 stores the newly computed offset 190 in the datastructure 180, e.g., in an element of the data structure 180 thatcorresponds to the X-Y-Z location of the device 130 when the acts 510and 530 were performed. If an offset value is already present in thiselement of the data structure 180, the server 170 updates the currentoffset to reflect an average of the current offset with the new one,optionally weighting the average as described above.

FIG. 6 shows an example method 600 for applying offset values stored inthe data structure 180. As above, the order shown is merely illustrativeand should not be construed as limiting.

At 610, device 130 reads its magnetic sensor 422 in the IMU 420 andprovides a measurement of local magnetic yaw 140 a.

At 620, the server 170 provides a current location 192 of the device130. For example, the TEKF 250 b tracks the location of the device 130based on input from the HMD 120, IMU 420, data structure 180, and ZVP260, computing each next location based at least in part on the currentone.

At 630, the server 170 performs a lookup into the data structure 180,e.g., using the current location 192 as X-Y-Z indices, and obtains theoffset 190 at the specified location. In some examples, the server 170also obtains offset values from neighboring locations, e.g., at adjacentindices or regions in the data structure 180.

At 640, the server 170 computes a weighted average of offsets, whichincludes the offset at the current location and the offsets of itsneighbors. This act may be skipped if averaging is not performed.

At 650, the server 170 adds the offset, which may be averaged, to thelocal magnetic yaw 140 a obtained at 610, to provide a corrected yawdirection 660. Such addition may be accomplished, for example, byoperation of the TEKF 250 b, which may apply the yaw offset in ameasurement function to enable the TEKF 250 b to track the device 130 inthe spatial FoR 108 b. With the corrected yaw direction 660 in hand, theserver 170 can accurately orient the device 130 in yaw, such that anyvirtual rays 132 from the device 130 align with the axis 138 of thedevice 130, thereby enabling the device 130 to be used as an accuratepointer.

FIG. 7 shows an arrangement similar to that of FIG. 1, but here the user110 shares the space 102 with a second user 110 a. The second user 110 awears an HMD 120 a, which may have its own depth camera 122 a, andcarries a device 130 a, which may be similar to the device 130. In theexample shown, the second user 110 a is outfitted with additionaldevices, such as a chest sensor 710 and an ankle sensor 712. Each ofthese sensors 710 or 712 may have its own Wi-Fi interface and its ownIMU, which includes a magnetic sensor, gyroscopes, and/oraccelerometers. The sensors 710 and 712 may thus have similarcapabilities in relevant respects to the devices 130 and 130 a. Forexample, each sensor or device may be capable of generating its ownmeasures of local magnetic yaw 140 a. Unlike the devices 130 and 130 a,however, the sensors 710 and 712 may not easily be visualized using thedepth camera 122 a. Accordingly, sensors 710 and 712 may be consumers ofoffsets 190 in the data structure 180 but need not be providers ofoffsets, as there may be no yaw references 140 b to be used forcomparison. The sensors 710 and 712 may still benefit from yawcorrection, however, by applying offsets 190 already stored in the datastructure 180 to their own measurements of local magnetic yaw direction140 a.

One should appreciate that yaw offsets 190 are not specific to anydevice or user, but rather are applicable to any device operated by anyuser. As the magnetic field 104 is quasi-stationary, offsets generatedfor one device may be applied to any other device. Rather thanreflecting properties of devices, the offsets 190 are intended toreflect properties of the space 102, which may be accessed forcorrection by any device that uses a magnetic field to orient itself.

Just as consumers of offsets 190 need not also be providers of offsets,neither do providers of offsets also need to be consumers. Some devicesor sensors may be both providers and consumers, however.

In some examples, the server 170 infers the pointing direction of adevice or sensor from other devices or sensors. For example, knowledgeof normal body mechanics may be applied to draw inferences about sensororientation.

Consider a case where user 110 a is walking forward (to the right in theperspective of the figure). It may be possible to infer the yawdirection of the chest sensor 710 and/or ankle sensor 712 based on aknown yaw direction of movement of the user 110 a. The server 170 maydetermine this yaw direction based on input from the HMD 120 a and/ordevice 130 a. The server 170 may then apply that direction of movement,which we call a “path tangent” 720, as a reference for other sensors.For example, the path tangent 720 may serve as a reference yaw 140 b,e.g. for the chest sensor, given that it provides an independent measureof yaw direction in the WCS FoR 108 b. In some examples, the server 170generates new offsets based on path tangents 720, which it applies asmeasures of reference yaw 140 b, and on local magnetic yaw 140 a asmeasured in the local magnetic FoR of the respective device. Operationproceeds in a similar manner to that shown in FIG. 5, except that pathtangents 720 are used as sources of reference yaw 140 b rather thanimages from the depth camera.

A top view 750 of user 110 a (bottom of FIG. 7) shows another exampleuse of path tangent 720. Here, it may be desired to provide an accuratemeasure of yaw direction of the ankle sensor 712, which is not easilyvisualized by the depth camera 122 a. To this end, the server 170 mayestimate a current path tangent 720 (D2) from chest sensor 710 or HMD120 a. At or about the same time, the accelerometer in the angle sensor712 may indicate a direction D3, based on the direction of force appliedby the user's ankle, and may further generate a local magnetic yaw 140a, having direction D1. The server 170 may then calculate a direction D3relative to the local magnetic FoR of the ankle sensor 712, and thensubtract D2 to produce an offset 190 of the ankle sensor 712 in thespatial FoR 108 b.

FIG. 8 shows an example method 800 that may be carried out in connectionwith the environment 100 and summarizes some of the features describedabove. The method 800 is typically performed, for example, by thesoftware constructs described in connection with FIGS. 2-4. The variousacts of method 800 may be ordered in any suitable way. Accordingly,embodiments may be constructed in which acts are performed in ordersdifferent from that illustrated, which may include performing some actssimultaneously.

At 810, a first measurement 140 a and a second measurement 140 b areobtained. The first measurement 140 a indicates an orientation of adevice 130 in a local magnetic frame of reference (FoR) 108 a based on amagnetic sensor 422 of the device 130, and the second measurement 140 bindicates an orientation of the device 130 in a spatial FoR 108 b basedon a separate instrument from the device 130, such as a depth camera 122or path tangent of HMD 120. The first measurement 140 a and the secondmeasurement 140 b are both made with the device 130 disposed in the sameknown location 192 and pointing in the same direction.

At 820, an offset 190 between the first measurement 140 a and the secondmeasurement 140 b is stored in connection with the known location 192,e.g., as an element in the data structure 180.

At 830, in response to the device 130 later being returned to the knownlocation, a corrected yaw direction of the device 130 is generated basedat least in part on a new measurement 140 a of the orientation of thedevice 130 in the local magnetic FoR and on the offset 190 stored inconnection with the known location 192.

An improved technique has been described for measuring yaw of a device130. The technique includes obtaining a first measurement 140 a of anorientation of the device relative to a local magnetic FoR and a secondmeasurement 140 b of the orientation of the device 130 relative to aspatial FoR, with both measurements made while the device is disposed ata known location and in the same orientation. The technique computes anoffset 190 between the two measurements and stores the offset inconnection with the known location 192. When the device 130 is laterreturned to the known location, the yaw direction 660 of the device isdetermined based on acquiring a new measurement 140 a of the device'sorientation in the local magnetic FoR 108 a and applying the offset 190as a correction.

Having described certain embodiments, numerous alternative embodimentsor variations can be made. For example, although certain functions havebeen described as being performed by the server 170, this description isillustrative rather than limiting, as the same functions could beperformed by the HMD 170 (acting as the server) or in some othercomponent. Although the server 170 is shown as a single component, oneshould appreciate that the server 170 may be implemented with any numberof computers and that at least some of the computers need not be local.

Section II: Device Velocity State Detection

This section presents an improved technique for detecting a velocitystate of a device, such as whether the device is moving, at rest, or ata dead stop. Embodiments of the technique presented herein may bepracticed in the environment of Section I but do not require yawmeasurements as described in Section I. The improvements of Section Iand Section II may each benefit from the other. For example, detectionof an at-rest state of a device may promote accuracy when generating yawoffsets 190. Similarly, detection of a change in yaw may alert the ZVP260 (FIG. 2) that a device that was previously in the dead-stop state isnow in motion. Particular features described as required for yawmeasurement may be optional for at-rest detection, and vice-versa.Therefore, a statement that any element is required in one sectionshould not be taken to mean that the same element is necessarilyrequired in the other section.

As described herein, an improved technique for detecting a velocitystate of a device includes generating multiple phase measurements foreach of multiple packets emitted by the device and monitoringdifferences between phase measurements made for different packets. Thetechnique further includes asserting a particular velocity state of thedevice based on a condition of the monitored differences. In someexamples, detecting the particular velocity state of the device cantrigger a correction for IMU drift.

FIG. 9 shows the example environment 100 of FIG. 1, with particularfeatures emphasized to depict aspects of velocity state detection. Inexample operation, the user 110 holds the device 130, and the device 130emits packets, such as Wi-Fi packets. The packets travel through space102 and impinge upon antenna arrays 160. For example, a packet 912 maytraverse paths 910 a, 910 b, and 910 c to antenna arrays 160 a, 160 b,and 160 c, respectively. Although shown as following simple paths, thepacket 912 spreads out from device 130 in all directions and typicallyfollows many paths to each antenna array 160. Multipath effects arisefrom reflections from walls, ceilings, floors, furniture, and so forth,and can be quite complex. Typically, each antenna of each antenna array160 receives the packet 912, and associated processing circuitrygenerates CSI (Channel State Information). The CSI typically containsinformation about the multipath signal received, such as its phase. Eachantenna array 160, in response to receiving a packet, sends the CSI 920a, 920 b, or 920 c of its constituent antennas to the server 170, wherethe ZVP 260 analyzes changes in the CSI across different packets anddetermines a velocity state of the device 130. Typically, each antennasends respective CSI, such that an antenna array 160 having fourantennas sends four sets of CSI.

Continuing with reference to FIG. 9, the ZVP 260 is configured to trackvelocity of the device 130, assigning the device 130 to one of thefollowing velocity states:

-   -   A Moving State, which indicates that the device 130 is in        motion;    -   An At-Rest State, which indicates that the device 130 is either        stopped or nearly stopped; and    -   A Dead-Stop State, which indicates that the device is at a        complete stop.

The dead-stop state may be regarded as a sub-state of the at-rest state,as a device that is at a dead-stop is also at rest. The two statesdiffer in their degree of certainty, however, with the dead-stop stateindicating a higher degree of certainty that the device is at a completestop than the at-rest state. To track the state of the device 130, theZVP 260 includes an at-rest detector 260 a, a dead-stop entry assessor260 b, and a dead-stop exit assessor 260 c. The at-rest detector 260 ais configured to determine whether the device 130 is in the moving stateor the at-rest state. The dead-stop entry assessor 260 b is configuredto detect when the dead-stop state has been entered. Both the at-restdetector 260 a and the dead-stop entry assessor 260 b operate primarily(and in some cases solely) based on Wi-Fi signals emitted from thedevice 130, optionally receiving additional input from the HMD 120, theOOEKF 250 a, and/or the VEKF 250 c. The dead-stop exit assessor 260 c isconfigured to detect when the dead-stop state has been exited, i.e.,when it can no longer be determined with a sufficiently high level ofconfidence that the device 130 is at a true dead stop. The dead-stopexit assessor 260 c preferably takes input from all relevant sources,including Wi-Fi, HMD 120, IMU 420, and EKFs 250 a and 250 c.

Although the environment 100 shows three antenna arrays 160 a, 160 b,and 160 c, the at-rest detector 260 a may work effectively with as fewas one antenna array 160. Performance may improve, however, withadditional antenna arrays 160, and three antenna arrays 160 appear to beoptimal in most cases. Preferably, each antenna array 160 includesmultiple antennas.

FIG. 10 shows an example antenna array 160 in additional detail. Theantenna array 160 may be representative of antenna arrays 160 a, 160 b,and 160 c. The antenna array 160 is seen to include an apex antenna1010A and multiple base antennas 1010B1, 1010B2, and 1010B3. Othernumbers of base antennas may be used. The antennas 1010 are preferablyspaced apart such that the distance between the apex antenna 1010A andeach of the base antennas is less than Pi radians at Wi-Fi frequency.Assuming 5 GHz Wi-Fi signals, Pi radians works out to approximately 3cm. Thus, the maximum distance between the apex antenna 1010A and eachbase antenna is preferably less than 3 cm. The actual distance ispreferably barely less than 3 cm, however, as making the distance muchsmaller might cause interference between antennas and degrademeasurement accuracy. The distance between different base antennas isnot critical, but it is preferably on the same order. Also shown in FIG.10 is a Wi-Fi processor 1020. The Wi-Fi processor 1020 uses a commonreference clock for generating CSI for all of the antennas 1010. Forexample, Wi-Fi processor 1020 includes a 4×4 Wi-Fi chip capable ofhandling the four antennas. Use of a common clock preventsdevice-frequency errors from arising between antennas 1010 in array 160.No effort need be made to synchronize clocks across antenna arrays 160,however. One should appreciate that the terms “apex” and “base” aremerely names that facilitate description and that different terms may beused for different types of antenna arrays.

FIG. 11 shows an example graph 1100 of unwrapped CSI phase versussub-channel frequency. In an example, data of the graph 1100 isgenerated by Wi-Fi processor 1020 in an antenna array 160 in response toone of its antennas 1010 receiving a wireless packet. In an example, theCSI of each antenna 1010 forms the basis for a respective graph 1100.The graph 1100 shows phase as measured by multiple Wi-Fi sub-channels,which are labeled 1-N, where “N” is the number of sub-channels. Someimplementations represent sub-channels from −N/2 to N/2. This is merelya convention. Multiple phase measurements 1110-1 through 1110-4(collectively, 1110) are shown, with the line in each illustrated squarerepresenting a phase measurement. A phase 1130 of the 0-th sub-channelis not directly measured but is rather inferred from other phase data.

It can be seen that the unwrapped CSI phase has a slope 1140 and ay-intercept 1150. The slope 1140 corresponds to the integer part of anumber of wavelengths traveled by the Wi-Fi signal from the device 130to the antenna 1010. The y-intercept 1150 corresponds to the fractionalpart of the number of wavelengths traveled. Thus, the slope 1140provides a course measure of distance traveled, whereas the y-intercept1150 provides a fine measure of distance traveled. The y-intercept 1150may vary between 0 and 2-Pi radians (or equivalently, between −Pi and+Pi radians).

Our experiments have shown that neither the slope 1140 nor they-intercept 1150 of the unwrapped CSI phase is consistent from onepacket to the next, even when distances are kept constant. The slopechanges on account of variable packet-detection delays found incommercially-available Wi-Fi devices, whereas the y-intercept changes onaccount of device-frequency offset. However, we have recognized thatdifferences in y-intercepts 1150 as measured between different antennas1010 of an antenna array 160 still provide useful information fordetecting whether a device is moving or at rest.

FIGS. 12a and 12b show an example arrangement for detecting a velocitystate of a packet-emitting device based on measurements of CSI phase byan antenna array 160. FIG. 12a shows an example antenna arrangement, andFIG. 12b shows an example method 1200. The at-rest detector 260 atypically performs the method 1200, with help from one or more antennaarrays 160, to effectively determine whether the device 130 is in theat-rest state or the moving state.

At 1210 of FIG. 12b , the antenna array 160 receives a packet, and eachof the antennas 1010 in antenna array 160 generates respective CSI forthat packet. From the CSI of each antenna 1010, a respective y-intercept1150 is identified, e.g., from the respective unwrapped CSI phase. Werefer to each y-intercept as a “theta” (Θ), with Θ_(A) (FIG. 12a ) beingthe y-intercept derived from apex antenna 1010A and Θ_(B1) throughΘ_(B3) being the y-intercepts of the respective base antennas 1010B1through 1010B3.

At 1220, the server 170 computes values of delta-theta (ΔΘ) asdifferences between Θ_(A) of the apex antenna 1010A and each of theΘ_(B)'s, as shown. The results are three ΔΘ's, which we call “sigmas”s), e.g., Σ₁, Σ₂, and Σ₃. By computing each ΔΘ (or Σ) value, the server170 acts to remove device-frequency offset, i.e. slight differences inWi-Fi frequencies between the packet-emitting device and the antennaarray 160. We may assume that all antennas 1010 in any antenna array 160share a common reference clock, although clocks between antenna arraysmay differ. Values of Σ are thus corrected for device-frequency offset.

At 1230, common-mode noise is removed by computing differences between Σvalues. For example, providing three Σ values (Σ₁, Σ₂, and Σ₃) meansthat there are three unique pairs of Σ values, {Σ₁, Σ₂}, {Σ₂, Σ₃}, and{Σ₂, Σ₃}, which we can use as a basis for computing ΔΣ's. At 1230, wecompute these three ΔΣ values, shown as ΔΣ₁, ΔΣ₂, and ΔΣ₃. These ΔΣ'sare also referred to herein as “gammas” (Γ's) with act 1230 producingthree Γ's: Γ₁, Γ₂, and Γ₃. At the completion of 1230 (or in paralleltherewith), operation returns to 1210, whereupon another packet isreceived and processed per acts 1210, 1220, and 1230.

Once two, preferably consecutive, packets from the device 130 have beenprocessed, operation proceeds to 1240, whereupon the server 170 computesΔΓ's between the two packets, P1 and P2. For example, ΔΓ₁ is thedifference between Fi for packet P2 and Fi for packet P1, and likewisefor the other ΔΓ's. The packets P1 and P2 may be separated in time byapproximately 20 milliseconds, a short-enough interval to allow smallchanges in velocity to be detected, but not so small that differencesare unlikely to be detected. Other time intervals between packets may beused, however.

Operation next proceeds to 1250, whereupon the method 1200 tests whetherany of the ΔΓ's computed at 1240 falls below a predetermined threshold.If so, at 1260 the device is at rest and the server 170 asserts theat-rest state 1280. If not, the at-rest detector may determine that thedevice is in the moving state 1290 (act 1270). In some examples,assertion of the at-rest state 1280 may be delayed until it isdetermined that at least one ΔΓ from among all those computed remainsless than the threshold for some number of packets. As an example, avariable may be set to an initial value when the device is moving. Thevariable may be decremented for each packet that produces at least oneΔΓ below the threshold and may be incremented for each packet thatproduces no ΔΓ below the threshold. The variable may be limited betweenminimum and maximum values. With this scheme, the at-rest state 1280 isasserted when the variable reaches the minimum value.

It should be noted that the at-rest state 1280 may be asserted usingonly a single antenna array, as described above, but performance mayimprove by using additional antenna arrays 160. Should multiple antennaarrays 160 be used, acts 1210 through 1240 of the method 1200 may beperformed for each of them. But rather than comparing, during act 1250,the ΔΓ's for a single antenna array 160 for detecting the at-rest state1280, act 1250 instead looks across all antenna arrays 160. For example,assuming three antenna arrays 160 are used, if any of the resulting nineΔΓ's computed across the three antenna arrays 160 is below thethreshold, the device is considered to be at-rest, even if the thresholdis exceeded by any or all of the other ΔΓ's across the antenna arrays160. Making the at-rest decision in this fashion reflects the fact thatmovement of the device usually affects CSI as measured by all antennaarrays 160 by a relatively large amount, whereas movement of a person orobject usually affects CSI as measured by only a subset of antennaarrays and to a lesser amount. If one or more of the ΔΓ's shows littleor no change, then probably the disturbance is not caused by devicemotion.

Although the at-rest detector 260 a is configured to determine whetherthe device 130 is moving or at rest, accumulated errors may cause theat-rest detector 260 a to assert the at-rest state when the device 130is moving very slowly. The at-rest detector 260 a is particularly robustagainst noise and is expected to improve with additional variance tuningof the extended Kalman filters 250. Although not a perfect dead-stopdetector, the at-rest detector 260 a has been shown to produce an errorof less than a few cm/sec (such as 7 cm/sec), using CSI-based techniquesonly. For purposes of at-rest detection, the device 130 is considered tobe “at-rest” if it is stopped or moving at a velocity less than thisspeed.

Detection of the at-rest state confers significant benefits, even if itis not true dead-stop detection. For example, the server 170 may use theassertion of an at-rest state to trigger operation of the dead-stopentry assessor 260 b, which is expected to detect a true dead-stop morereliably than the at-rest detector 260 a. Also, detection of an at-reststate may allow the server 170 to adjust inputs to the extended Kalmanfilters 250, to best tailor their operation for current circumstances.For instance, the server 170 may apply a zero-velocity measurementfunction to the TEKF 250 b in response to an at-rest detection andprovide the TEKF with an increased variance, so as to reflect a lowerlevel of confidence that the velocity is actually zero. When operatingin the dead-stop state 1280, the server 170 may provide the samemeasurement function to the TEKF 250 b, but with a very small variance,so as to reflect higher confidence that the velocity actually is zero.Both the TEKF 250 b and the VEKF 250 c may update their respectivemeasurement functions each time the at-rest detector 260 a detects theat-rest state 1280. In addition, and given that the TEKF 250 b tracksdevice location, improving the accuracy of the TEKF 250 b promotes moreaccurate measures of device location. In so doing, the ability to detectthe at-rest state 1280 improves yaw measurements (Section I). Theoverall effect of at-rest detection is thus to leverage knowledge ofzero or near-zero velocity to increase accuracy of both tracking and yawmeasurements.

One should appreciate that any movement of the device 130 within thespace 102 changes the CSI phase of signals received by the antennaarrays 160. Typically, device movement causes all path lengths andtherefore all CSI phase values to change for all antenna arrays 160. Incontrast, changes in the environment, e.g., caused by people or objectsmoving around, tend to affect CSI phase much more for some antennaarrays 160 than for others. For example, the antenna array 160 that isclosest to the moving person or object is likely to be affected morethan antenna arrays 160 that are further away, especially if the personor object is located between the device and the closest antenna array.Given this difference in phase behavior between a moving device and amoving person or object, the server 170 is able to differentiate betweenthe two cases based on how CSI phase changes from the different antennaarrays 160. In particular, one can say whether the device 130 is merelyat rest or has entered a dead-stop state by applying differentthresholds for phase stability, as measured by the different antennaarrays 160.

In an example, the dead-stop entry assessor 260 b (FIG. 9) receives thesame delta-sigmas (ΔΣ's) that are used by the at-rest detector 260 a inmethod 1200. Rather than analyzing differences in ΔΣ's between twoconsecutive packets, however, the dead-stop entry assessor 260 b looksfor stability in ΔΣ's over multiple consecutive packets, such as threeor more packets, for example, which may arrive over an interval of 60 ormore milliseconds (assuming 20-millisecond packet spacing). Preferably,the dead-stop entry assessor 260 b looks at ΔΣ's from all antennas 1010of all antenna arrays 160 that are in use, e.g., nine antennas 1010 fora three-array setup. In an example, the dead-stop entry assessor 260 buses a combination of deviation thresholds and voting to determinewhether the device 130 should enter the dead-stop state. Accordingly,the dead-stop entry assessor 260 b may work solely on the basis of Wi-Fisignals (CSI). However, some embodiments augment CSI with input fromother sources, such as IMU 420, EKFs 250, and HMD 120.

Operation of the dead-stop exit assessor 260 c is generally simpler thanthat of the dead-stop entry detector 260 b, as any significant change inposition or orientation of the device from any source (e.g., IMU 420,HMD 120, CSI, or yaw) can cause the device to exit the dead-stop state.Also, certain embodiments employ a more conservative standard forentering a dead-stop state than for exiting it. Given that some or allof the EKFs 250 may be tuned differently when the device 130 is in adead-stop state than when it is in the other states, optimal performancemay depend on not being wrong when declaring the dead-stop state.However, the consequences in terms of user experience of wronglydeclaring that a device is not in the dead-stop state are typically lesssevere.

Given the reliance of certain embodiments on CSI for asserting thevarious states (moving, at-rest, and dead-stop), accurate performancemay depend on the CSI being valid. In some examples, validity of CSI isverified through the use of packet bursting. For example, Wi-Fi settingsof the device 130 may be configured to disable packet aggregation andtherefore to permit packet bursting. Enabling or disabling aggregationis typically a device-driver setting of the Wi-Fi component, but detailsof the setting may vary across manufacturers. We have experimentedsuccessfully with disabling packet aggregation using Wi-Fi devicesobtained from Laird Technologies, Inc., of Chesterfield, Mo. Bydisabling Wi-Fi packet aggregation, the device 130 is able to sendseparate packets in very quick succession, on the order of once every200 microseconds. In an example, all the packets within each burst carrythe same payload. Bursting thus provides redundancy at high speed, whichenables the ZVP 260 to operate more robustly. Different bursts,conveying different payloads, may be sent approximately every 20milliseconds.

We have recognized that multipath characteristics of the space 102 areunlikely to change by measurable amounts within the span of a singleburst, which may last only a few hundred microseconds or less. The space102 typically remains stable within that timeframe. Any large change inCSI within a burst then almost certainly indicates a hardware error orother anomaly. If a burst contains two packets for which CSI differs bymore than a threshold amount, the server 170 may compare the CSI of thetwo packets with the CSI of a packet from an immediately previous burst.If the CSI of one of the two packets from the current burst matches theCSI of the packet from the previous burst to within a thresholddifference, the other of the two packets from the current burst isdiscarded as erroneous. If the CSI of both packets of the current burstdiffers from the CSI of the packet from the previous burst by more thana threshold, all the packets of the current burst may be discarded.Discarding packets that convey erroneous CSI prevents that CSI fromdegrading the quality of velocity-state detection. In an example, theCSI features that are the subject of the above-described comparison arethe delta-sigma (ΔΣ) values, as described in connection with FIG. 12 b.

FIG. 13 shows in greater detail certain components of the server 170that support both yaw measurement (Section I) and velocity-statedetection (this Section II). Operative connections are shown among theEKFs 250, the ZVP 260, the yaw processor 172, and the data structure180.

To support yaw measurements, the orientation-only EKF (OOEKF) 250 areceives input from the IMU 420 in the device 130. The input includesmeasures of local magnetic yaw 140 a, i.e., measurements of yaw in thedevice's local magnetic frame of reference (FoR) 108 a (FIG. 1). OOEKF250 a processes the input 140 a to generate output 140 a 1, whichprovides a processed version of the local magnetic yaw 140 a. As OOEKF250 a tracks only orientation of the device 130, output 140 a 1 providesa very stable view of device orientation in the local magnetic FoR 108a.

At or about the same time that OOEKF 250 a is processing a measurementof magnetic yaw 140 a, the tracking EKF (TEKF) 250 receives andprocesses a measurement of reference yaw 140 b, e.g., from the HMD 120(FIG. 1). From this input, the TEKF 250 b generates an output 140 b 1,which provides a processed version of the reference yaw 140 b. In anexample, the TEKF 250 b uses an orientation measurement function intowhich the offset 190 from the data structure 180 is applied, enablingthe TEKF 250 b to track the device 130 in the spatial frame of reference108 b.

As shown, the TEKF 250 b also receives input from IMU 420, offsets 190from data structure 180, and velocity state 1320 from ZVP 260. It mayfurther receive additional spatial inputs from the HMD 120. Based on thereceived information, TEKF 250 b generates a location estimate 192 a ofthe device 130, as well as an output for device pose 1310, whichincludes both device location and orientation. In some examples, thedata structure 180 receives the location estimate 192 a as the location192 of the device 130 and uses that location estimate 192 a for storingand/or retrieving offsets 190. The TEKF 250 b is continually computingthe next location estimate 192 a based at least in part on the currentlocation 192.

In some examples, the server 170 adjusts the variance of the TEKF 250 bbased on the nature of the offset 190 that is applied to the TEKF'sorientation measurement function. For example, the server 170 sets ahigh variance (lower confidence) for an offset 190 that is based on asingle image acquired from the HMD 120, whereas the server 170 sets alower variance (higher confidence) for an offset based on an average ofmany samples. In some cases, the variance may scale with the count 190 a(FIG. 2) of samples averaged together at the current location 192.

Continuing with reference to FIG. 13, the yaw processor 172 receivesboth the processed yaw 140 a 1 (from OOEKF 250 a) and the processed yaw140 b 1 (from TEKF 250 b). From these inputs, the yaw processor 172generates an offset 190, which may be stored in the data structure 180in connection with a corresponding location estimate 192 a.

As further shown, the velocity EKF (VEKF) 250 c provides input to theZVP 260 for assisting in the determination of velocity state 1320. In anexample, the VEKF 250 c tracks the full pose (location and orientation)of the device 130 in the local magnetic frame of reference 108 a.Significantly, position and velocity as tracked by the VEKF 260 c areboth biased toward zero in all directions. For example, the VEKF 250 cis tuned to detect changes in position and/or velocity from a quiescentstate of all zeroes. The VEKF 260 c then uses a zero-velocitymeasurement function to drive the velocity to zero. The variance used bythis measurement function depends on the velocity state 1320. As withthe TEKF 250 b, variance is smaller in the dead-stop state and muchlarger in the at-rest state. In an example, the dead-stop exit assessor260 c monitors output from the VEKF 250 c, e.g., in the form ofposition, velocity, and in some cases accelerometer bias of IMU 420. Thedead-stop exit assessor 260 c then exits the dead-stop state if theoutput changes by more than a threshold amount, as such changes indicatethat the device 130 has started to move.

In the depiction of the ZVP 260, dashed lines indicate optionalconnections. Preferably, the dead-stop exit assessor 260 c uses inputsfrom all sources: OOEKF 250 a, VEKF 250 c, and CSI (Gammas) 1232, aswell as input from HMD 120, magnetic sensor 422, other IMU output, andthe like. The at-rest detector 260 a and the dead-stop entry assessor260 b both rely upon Gammas (F's) 1232, but may each receive additionalinput from the various sources to assist with their respectivefunctions.

With the depicted arrangement, the server 170 may correct for drift inthe IMU 420. For example, detection of an at-rest or dead-stop state cantrigger a re-referencing of location as measured by the IMU 420 to thecurrent estimate 192 a of device location, e.g., as provided by the TEKF250 b. Such correction improves the trustworthiness of output from theIMU 420, at least in the short-term, as a determinant of location of thedevice 130. Detections of at-rest and/or dead-stop states are expectedto occur frequently. Thus, consequent corrections of IMU drift can keepthe IMU 420 generally accurate over time. Detections of both the at-restand dead-stop states may trigger IMU correction, with dead-stopdetections expected to produce more accurate corrections than at-restdetections on account of the generally more accurate location estimates192 a from the TEKF 250 b during dead-stop than during at-rest. Inaddition to providing opportunities to correct for IMU drift, detectionsof at-rest and/or dead-stop states also allow the server 170 to averageposition estimates from multiple sources to build up highly accurateposition values.

Given that the device 130 may be provided as a virtual pointer, such asthe one described in incorporated U.S. patent application Ser. No.15/655,489, it is essential for best user experience that orientation ofthe device 130 be measured accurately, as even small pointing errors canproduce unsatisfactory results. It is also crucial to user experiencethat the velocity state 1320 of the device 130 be determined with asmuch certainty as possible.

To these ends, FIG. 14 shows a particular arrangement of components ofdevice 130 that helps to promote accurate readings of both orientationand velocity. One should appreciate that the depiction of device 130 inFIG. 14 is merely illustrative of relevant features. No effort has beenmade to show the device 130 as it might eventually appear in acommercial product.

Here, the device 130 has a handle or hand grip 1420. A user (not shown)positioned to the left of the figure, might be expected to hold thedevice 130 with fingers wrapped around the grip 1420 and with the user'shand extending to the right. With this arrangement, the IMU 420 in thedevice 130 aligns approximately with the location of the user's wristjoint, such that the IMU 420 remains approximately stationary as theuser pivots the wrist (assuming the user is otherwise stationary). Inthis manner, changes in orientation of the device 130 can be measuredwith a minimum velocity component.

The device 130 also includes an antenna 1410, which sends Wi-Fi packetsfrom which the above-described CSI is generated. The antenna 1410,although functioning as a packet transmitter, may nevertheless beregarded as a velocity sensor, given that the server 170 determines thevelocity state 1320 of the device 130 based on packets emitted by theantenna 1410.

As shown, the antenna (or velocity sensor) 1410 is located at theextreme end of the device 130, where it typically moves more than anyother part of the device 130 in response to wrist rotation by the user.The antenna 1410 is thus optimally positioned for sensing velocity ofthe device 130, as any rotation of the user's wrist is amplified overthe distance to the antenna 1410.

Although the depicted arrangement of components in FIG. 14 may beregarded as optimal, it is certainly not required. Other embodiments mayuse different arrangements.

FIG. 15 shows an example method 1500 that may be carried out inconnection with the environment 100 and summarizes some of the featuresdescribed in this section. The method 1500 is typically performed, forexample, by the software constructs described in connection with FIGS.2-4. The various acts of method 1500 may be ordered in any suitable way.Accordingly, embodiments may be constructed in which acts are performedin orders different from that illustrated, which may include performingsome acts simultaneously.

At 1510, phase measurements are generated for each of a set of packetsemitted by the device 130. In an example, such phase measurements aretheta values (Θ) based on CSI or other readings from the antenna arrays160.

At 1520, differences are monitored between phase measurements made ofdifferent packets emitted at different times. For example, the at-restdetector 260 a computes delta-theta (ΔΘ) values, which removedevice-frequency offset, and computes delta-sigma (ΔΣ) values, whichremove common-mode noise. Differences between ΔΣ values (ΔΓ's) are thendetermined between packets arriving at different times, such as 20milliseconds apart.

At 1530, a particular velocity state 1320 of the device 130 is assertedbased on a condition of the monitored differences. For example, theat-rest state 1280 may be asserted if any of the ΔΓ's falls below athreshold (FIG. 12b ). Alternatively, the moving state 1290 may beasserted if all of the ΔΓ's exceed the same threshold. The dead-stopstate may be asserted if the dead-stop entry assessor 260 b detectsstability in ΔΣ's over multiple packets.

An improved technique has been described for detecting a velocity state1320 of a device 130. The technique includes generating multiple phasemeasurements (e.g., 0 values) for each of multiple packets 912 emittedby the device 130 and monitoring differences (e.g., ΔΘ's and ΔΣ's)between phase measurements made for different packets. The techniquefurther includes asserting a particular state 1320 of the device 130based on a condition of the monitored differences.

Having described certain embodiments, numerous alternative embodimentsor variations can be made. For example, although certain velocity stateshave been shown and described, embodiments hereof may include a greateror lesser number of velocity states, or other velocity states notspecifically described, such as a high-velocity state, a spinning state,a medium-velocity state, and the like. Further, although a computationalapproach based on CSI has been described, embodiments hereof may workwith phase measurements made using other approaches, such astime-of-arrival, time-of-flight, mixed time-of-flight, angle-of-arrival,and the like. Further, although the illustrated embodiments make noassumption about packet-detection delays in Wi-Fi devices, the approachas described herein may be used advantageously in the event that Wi-Fimanufacturers start providing devices with constant (or consistent)packet-detection delays. Indeed, embodiments under such circumstancesmay be simpler that those illustrated, as constant packet detectiondelays may obviate the need to correct for device-frequency offset.Also, although detecting velocity state of a device has been describedin connection with correcting for IMU drift, this is merely an example.Other uses of velocity detection include improving yaw measurement and,more generally, improving user experience by enabling a game or otherapplication to represent movement of a device or user morerealistically.

Section III: Locating Objects in a World Coordinate System (WCS)

This section presents a technique for locating objects in a WorldCoordinate System (WCS). The technique described in this section may bewith the embodiments described in Sections I and II, but it may also beused independently.

Existing AR systems use computer vision to create a spatialunderstanding map of the user's environment and create spatial anchorsfor hologram placement. Each spatial anchor is the origin of acoordinate system, and its position is adjusted over time relative toother spatial anchors in the user's environment. The purpose of spatialanchors is to anchor holograms in 3D space and re-localize the hologramto adjacent spatial anchors if the user moves the hologram throughspace. Each spatial anchor is also a point in every other spatialanchor's coordinate system, thus establishing what can be considered anoverall coordinate system. However, there are two problems with spatialanchors:

-   -   The relationship between spatial anchors is imprecise.    -   The relationship between spatial anchors shifts continually with        time.

These problems produce some side effects:

-   -   1. System errors in hologram placement can be especially large        when the hologram is not close to the user.    -   2. Application developers may need to expend a lot of extra        effort to create applications such that holograms will be        attached to multiple anchor points.    -   3. Application developers may need to accept significant limits        on resource availability because spatial anchor management        (creation and maintenance by the system) is computationally        intensive.

An additional responsibility of computer vision systems for AR is toidentify when a user is gazing at a hologram that is outside of theuser's field of view (behind a wall or other physical object) so thatthe hologram should rightly be hidden from view or occluded. Due to theaforementioned problems with spatial anchors, occlusion often workspoorly for rendering holograms that are not near the user. In addition,enabling hologram occlusion is computationally intensive for thecomputer vision system (because it would need to continually scan auser's environment to provide hologram placement with occlusion as afeature set). Consequently, application developers are cautioned to usethe feature set sparingly.

What is needed is a stable Cartesian World Coordinate System (WCS) thatcould describe an entire home (and adjacent areas), a commercialbuilding, or a corporate campus, including walls, floors, ceilings, andfurnishings. The WCS could be used to realize the system described inU.S. Patent Publication No. 2018/0061127, filed Jul. 21, 2017, andentitled “Managing Virtual Content Displayed to a User Based on MappedUser Location,” the contents and teachings of which are incorporatedherein by reference. For AR applications, it would provide thefollowing:

-   -   Minimize hologram placement and occlusion errors inherent in        current AR systems.    -   Enable the use of techniques to improve the user's visual        perception of hologram location.    -   Simplify the process of re-localizing (the holograms of)        multiple users within the same physical environment.    -   Reduce runtime computational load to free additional resources        for game developers.    -   Free game developers to create applications that take full        advantage of the entire physical environment without cumbersome        management of spatial anchors.

Finally, body tracking is an example that highlights the need for a WCS.If a player's body movements were being tracked through space to createa holographic avatar, there would be a problem in existing systems atthe boundaries between local coordinate systems (those relative to twodifferent anchor points). Position errors would create a highlydistorted avatar while passing through the boundary because differentparts of the body would be tracked relative to different coordinatesystems. With a single WCS, there are no such boundaries.

Solution

Today, AR system vendors enable multiple players in the same physicalenvironment to share spatial understanding maps so all players can seethe same holograms. Third party AR Cloud providers enableinteroperability (between AR systems from different vendors) by creatingan additional computer vision layer on top of what the AR system vendorprovides. AR Cloud systems take a bottoms-up approach to these matters.

By contrast, Reavire takes a top-down approach.

-   -   Abstractly defines a World Coordinate System (WCS).    -   Each Device Family synchronizes to this world coordinate system        (e.g., HoloLens, Magic Leap, etc.)

The WCS is a fundamental part of the Reavire AR gaming system because:

-   -   Avatars depend on WCS to avoid working errors in avatar        movement.    -   Since the LPS is tied to avatar generation, the LPS can be        trained to the WCS as an error bound on the generation of the        avatar.    -   Reavire is empowering AR game developers to use standard 3D game        engine development processes unencumbered by spatial anchor        management or bandwidth-intensive map sharing.

LPS as Independent Observer

We use a Local Positioning System (LPS) as an independent observer. Asuitable example of an LPS is disclosed in PCT Publication No.WO/2018/039339, the contents and teachings of which are incorporatedherein by reference. The physical wireless access points (WAPs) or otherphysical markers define an abstract coordinate system that is the WCS.Augmented reality HMDs or mobile phones are synchronized to the WCSusing the defined physical markers and Relocation Anchor Points (RAPs),system-created and managed anchor points as defined in the Key Termssection below. With Relocation Anchor Points we eliminate the need fordevice-specific anchor points. The basic operational characteristics ofthese RAPs can be introduced as follows:

-   -   Any one device out of a device family can synchronize to the WCS        on behalf of all members of that device family.    -   Each device family synchronizes to the WCS independently.    -   Once devices in a device family are synchronized, communication        about location occurs in the WCS layer. How this WCS layer        eliminates the need for device-specific anchor points will be        further discussed below.    -   This allows multiple device families (e.g., HoloLens or Magic        Leap) to play in the same game.

Furthermore, the LPS system bounds errors and helps prevent errorconditions typical of computer vision systems (e.g., low light, brightlight, ghosting, dark corners, visually indistinct environments).

The WCS as Independent Layer

The Reavire WCS is an independent layer on top of the underlyingcomputer vision layer within the AR systems. By decoupling these layers,we provide a stable coordinate system interface and free game developersfrom anchor point management.

The standard primitives needed to synchronize to and build the WCSinclude:

-   -   1. Anchor Points: Any AR system that aims to tie content        (holograms) to real space will support the concept of anchor        points.    -   2. Downloadable geometry: Any AR glasses that are designed to        work with third party game engines such as Unity must be able to        download mesh models of the observed geometry. These mesh models        are derived from but not equivalent to the spatial understanding        maps. Mesh models can be anything from surface meshes to higher        order geometry derived from these surface meshes. Uploading        geometry to the glasses is the same as uploading any hologram to        the glasses and, hence, always supported.    -   3. Observed Location of Headset: This is preferably a        requirement for all applications.    -   4. Observed Location of Real Objects: Application developers        expect support for tools such as Vuforia to locate real objects,        so all AR vendors support the ability to visually identify a 3D        object.    -   5. Low level networking control: The LPS system may require that        packets be sent on a specific channel separate from the        data-carrying channel. Channel selection support is standard for        WiFi. Even if the device cannot support our LPS protocol for        some reason, we can work around this by adding an extra sensor        clipped to the headset.    -   6. Time Stamps: These time stamps need to be on the order of        milliseconds in order to measure human movement. This is well        within normal standards.

With the support of AR system vendors, additional functionality can beimplemented to make the overall WCS more robust:

-   -   SLAM Input: Our LPS system is an independent observer that can        locate the headset (for example, to 6 cm accuracy). Examples of        benefits (that would be difficult or impossible for the AR        glasses to solve on their own) are as follows:        -   a. Show Virtual Content During Periods of Tracking Loss: The            gaming experience would be greatly improved by enabling the            glasses to continue to show Virtual Content during periods            of loss of visual tracking. There can be many causes of            tracking loss (e.g., low light, bright light, dark corners,            lack of distinct geometry). One example is covered by            PCT/US2018/020888, filed Mar. 5, 2018 and entitled            “Maintaining Localization and Orientation of Electronic            Headset After Loss of SLAM Tracking”, which is incorporated            herein by reference. In that example, the glasses would use            a combination of IMU, texture data from the nearby wall, and            the LPS system to perform image stabilization. In other            examples, LPS data alone is sufficient.        -   b. Disambiguation of Visually Similar Spaces: The LPS system            enables the glasses to disambiguate visually similar spaces.        -   c. Eliminate Ghosting: Ghosting can occur when a room is            initially mapped; then, in the process of mapping the space,            tracking errors build up as a larger space is mapped. Then,            when visiting the initially mapped room, the tracking errors            are large enough that the AR system remaps the room again.            The prior map remains as a “ghost” in an apparently            different space determined by the accumulation of tracking            errors. Ghosting can be eliminated by the AR system using            the LPS as a secondary comparison point to bound tracking            errors.        -   d. Search Scope Reduction: Re-localization algorithms based            on visual data alone are usually fast enough to satisfy the            requirements of the user. However, using LPS as an            additional input reduces the scope of the search and, hence,            reduces the total amount of compute power used. AR devices            are generally limited in compute power so any improved            efficiency frees up resources an application developer can            use. Battery life is also an important consideration.

Note that LPS training does not need to be complete for game play tostart. Only minimal training is normally required at the start of gameplay.

A major benefit of the WCS is that anchor points do not need to beexposed to game designers. Game designers can use a single stablecoordinate system for every scene of their game and will not have toworry about the complication of which anchor point to attach a gameobject to in order to make it appear stable to the users. This permitsgames running on the Reavire network to place content anywhere withinthe user's environment without needing to understand the precisegeometry of that environment (i.e., the game engine may place a hologramin a location that no device is currently viewing and that locationcould have been rearranged from the last time a device has seen it).

Key Terms

The following key terms are used in this document to describe examplefunctionality:

-   -   1. World Coordinate System (WCS): The world coordinate system is        defined by the placement of fixed identifiable objects within        the house (for example wireless access points used for the LPS).        A three-dimensional Cartesian coordinate system involves an        origin and two direction vectors.        -   a. One fixed object is defined as the origin.        -   b. Gravity defines the first direction vector as up/down.        -   c. The projected line onto a horizontal plane between two            fixed objects defines the second direction vector. This does            preferably require that the two objects not be placed in a            direct vertical line. The larger the horizontal spread, the            more accurate the direction definition will be.        -   The World Coordinate System is fixed and distinct from any            particular device's world coordinate system. We distinguish            the two by simply referring to the latter as the device            coordinate system, e.g. device, HoloLens, Magic Leap, or            Unity coordinate system.    -   2. Coordinate System Synchronization: This is the process of        synchronizing the device coordinate system to the World        Coordinate System (WCS.) The precise method depends on the exact        capabilities of the device. We cover example detection and        calibration procedures in detail below.    -   3. Mapping: This is the process of making a three-dimensional        virtual model of the house in the World Coordinate System (WCS).    -   4. Relocation Anchor Point: A Relocaiton Anchor Point (RAP) is a        detectable point in real space that is attached to a physical        location. It is labeled with a position (x, y, z) in the WCS and        it has an orientation that is an estimate of the WCS's        orientation. The RAP provides an estimate of the WCS that is        valid locally. The RAP is labeled. A device observes a RAP and        allows translation of the device coordinate from the device's        coordinate system into the WCS for use by the game engine (game        logic) dynamically during game play.    -   5. PlayZone Definition: The PlayZone is a simplified subset of        the virtual model of the house. See the above-referenced U.S.        Patent Publication No. 2018/0061127 for a full description of        the PlayZone.

Part A: Creation and Management of the WCS

Existing State of the Art: Anchor Points

Anchor points provide a method for a local coordinate system to beattached to a specific physical point or area. The preciseimplementation of an anchor point is not material for our discussion.Roughly speaking, an anchor point is a piece of point cloud map bigenough to permit the anchor point to be accurately placed within thatpoint cloud map. An origin and orientation vectors are algorithmicallygenerated from the anchor point data itself. In this manner, a localcoordinate system can be attached to virtually any physical location.

As part of the Simultaneous Location and Mapping (SLAM) system, eachpair of AR glasses is responsible for generating its own point cloudmap. Two pairs of glasses share a local coordinate system by sharing ananchor point. Although the anchor point data was created by one of theglasses, the algorithms for placement of the anchor point within anindependently generated map are good enough for a second pair of glassesto place the anchor point very close to the position and orientation ofthe first. With a little consideration, it is easy to see that after ananchor point is generated, whether it is placed in a map by the first orsecond user is immaterial. The same types of errors will occur. Thereason for this is that the SLAM system is continuously updating itsmap. Therefore, after some time, both devices will have evolved theirunderstanding of the space. Hence, the anchor point will slightly changeposition and orientation within these evolved maps. It is particularlythe slight change in orientation that causes the validity of the anchorpoint's coordinate system to be spatially limited. The lever arm effectof angular error quickly adds up to a noticeable error. Position erroris typically, but not always, small. Positional error is also constantover the area of interest. Hence, positional error of the placed originis typically not a large source of noticeable error in the associatedcoordinate system and hence placement error of associated holograms.

The following facts about anchor point should be considered when usingthem to anchor holograms to a physical location.

-   -   Anchor points are periodically repositioned as the device's        understanding of the world changes.    -   A device may reposition an anchor point even if the anchor point        is not presently in view of the device.    -   A device constantly locates itself in the environment. Hence, if        an anchor is not in view but an associated anchored hologram is,        then an equivalent amount of hologram positional error will        occur even if the anchor point itself is not repositioned.

Devices such as the Microsoft HoloLens attempt to provide certainbenefits within the limitations of the device. In the HoloLens, a“master” anchor point exists called the sharing stage. The HoloLensstores metadata for user-generated anchor points that permit thoseanchor points to be roughly placed before the space has been mapped. Thesharing stage anchor point provides the rough estimate of the globalcoordinate system. Although this does not provide a world coordinatesystem, it does allow a new user to see all holograms even if they havenot mapped the entire playable area yet. For room-sized games, a singleanchor point may be sufficient, and, in this case, a single coordinatesystem covers the entire area of interest.

In outdoor environments, GPS is combined with anchor points to give theillusion of a global coordinate system. If GPS can locate the devicewithin the range of validity of an anchor point, this method functions.The extent of validity of an anchor point is several meters, whichroughly corresponds to the accuracy of GPS. The word “illusion” is usedbecause the location of a device in this coordinate system is composedof the tuple, GPS coordinate, Anchor Point ID, and Local Coordinate.Orientation in this coordinate system can use gravity and magnetic northas direction vectors, and/or the local orientation of the anchor point.

Reavire WCS

A world coordinate system is the logical equivalent of using a singleperfect anchor point to provide a stable, uniform, accurate, and precisecoordinate system available throughout the entire playable area. By“perfect” we mean one that has no position or orientation errorassociated with it. Since perfect anchor points do not exist, theReavire WCS is constructed in a different manner.

The Reavire WCS is abstractly defined and tied to the real world so thatany device may synchronize to it. As mentioned, device families worktogether so that the entire synchronization procedure does not need tobe repeated by each member of the family.

The Reavire WCS is constructed independently for each device family thatconnects to it. The construction process is broken into two main phases:

-   -   Phase 1. The synchronization phase where a device in the device        family synchronizes to the abstractly defined WCS.    -   Phase 2. That same device continues to build a network of        Relocation Anchor Points (RAPs). The purpose of this network is        to provide a robust framework for device position and        orientation correction with respect to the WCS.

Synchronization to the WCS

Every gaming location is different—different layout, differentfurniture, etc. The one constant that can be counted upon is thepresence of Reavire Wireless Access Points, WAPs. These WAPs are used todefine the coordinate system for the location. This process describes atypical home installation of three WAPs, but very large installationswith hundreds of WAPs are possible. In cases with more than three WAPs,three adjacent ones would be selected for initial training, then thesame process used to train the third WAP would be repeated for theremainder. One WAP is defined as the origin. The pair of WAPs with thelargest horizontal plane distance between them is used to define theNorth vector. The second orientation vector is gravity and is providedby each of the devices. The WAPs are enhanced with registration marksand utilize line lasers to project a visible pattern that is easilyvisible by an AR device from far away.

A device uses machine vision object recognition techniques to locateeach WAP in the device's local coordinate system. The device makesrepeated measurements and is required to visit each WAP at least twiceto ensure that the positions of the WAPs are detected in the device'slocal coordinate system with sufficient accuracy and precision. That is,a WAP's location is measured and then remeasured to ensure that therelative position of three WAPs are accurately known. The number ofiterations required for convergence of this process is primarilydetermined by the quality of the SLAM system of the device. Thefollowing summarizes the process of synchronizing to the WCS.

-   -   1. User turns on device and is instructed to locate one of the        WAPs. Which WAP is immaterial; the system has no ability to        guide the user at this point. For notational purposes we call        this WAP0.    -   2. The user visually captures the position of WAP0 using machine        vision object recognition techniques.    -   3. The device creates an anchor point associated with WAP0. We        call this Anchor0. Now we define three different positions.        WAP0.p is the position of WAP0 in the local coordinate system of        the device, as measured by the vision system. Anchor0.p is the        position of Anchor0 in the device's local coordinate system.        -   a. Anchor0 is uniquely identifiable by the device.        -   b. At time of creation of Anchor0, the relative position of            Anchor0.p to WAP0.p is defined at this point. We call this            relative position Delta0. The device may drift causing both            positions to drift in the device's local coordinate system.            Anchor0 may also shift, but this can be differentiated from            device drift using machine vision location of WAP0 and the            fact that Delta® is fixed.        -   c. The above two facts imply that every time the device            returns to the proximity of Anchor0 the device's drift can            be measured and accounted for. This is explained in detail            in a subsequent section.    -   4. The user is instructed to walk to another WAP. We will call        this WAP1. As with WAP0 we create Anchor1, and measure both        WAP1.p and Anchor1.p. Hence Delta1 is known and fixed at this        point.    -   5. The user is then instructed to walk the third and final WAP.        We call this WAP2. Once again, we create an anchor, this time        Anchor2 and take measurements Anchor2.p and WAP2.p thus defining        Delta2.    -   6. The system determines the role of the various WAPs. The two        WAPs with the furthest horizontal separation are used to define        the North direction vector. One of these two will arbitrarily be        made the origin of the WCS. Each of the anchor points is given a        label, Anchor0.1, Anchor1.1 and Anchor2.1, indicating its        position in the WCS. The initial values of the labels are as        follows. For simplicity of presentation, we assume Anchor0 is        the origin and Anchor1 defines the North vector.        -   a. Anchor0 Anchor0.1 is set (0,0,0). This label does not            change over time.        -   b. Anchor1: Anchor1.1 is set to (0,0,r) where r is the            initial estimate of the distance between Anchor0 and            Anchor1. We use a left hand coordinate system with y            pointing upward. During the WCS definition process only z            coordinate may change. The zeroes are immutable by            definition. Anchor1 lies on the z-axis of the WCS coordinate            system. The North vector is a direction vector parallel to            the z-axis.        -   c. Anchor2: Anchor2.1 is set to (x, y, z) which is the            initial estimate of the WCS position after definition of the            origin and North vector. These values will likely move a lot            during the synchronization process to the WCS.    -   7. The user is instructed to go the origin. Once Delta® is        measured to be stable, the origin is considered stable and the        user's position is calibrated.    -   8. The user is instructed to go to Anchor1 to define the North        vector. Measurements are taken such that Delta1 is calculable.        When Delta1 is stable and Anchor1 correctly identifies WAP1.p,        the distance from Anchor0 to Anchor1 is calculated. The process        of going back and forth between Anchor1 and Anchor0 is repeated        until the error in the distance between them is sufficiently        low. The distance is measured in the local coordinate system of        the device. The device's drift is measured at each return to the        origin and subsequent measurements are taken with the new        drifted starting value. This is important because the correction        process ensures that the device accurately places Anchor0 in its        local coordinate system, which we assume is subject to an        arbitrary amount of drift. The system may use a running average        of the distance measurement to determine when the Anchor0 to        Anchor1 distance is sufficiently calibrated.        -   a. Anchor1.1 is now set and remains immutable unless the            user restarts the WCS synchronization process. The WCS is            defined at this point; however, the synchronization            procedure is not complete. The system calibrates the North            vector by calculating the position of Anchor1 in the            coordinate system of Anchor0 and then using the result to            calibrate the North vector with respect to the local            coordinate system of the device. Anchor1 may now be used to            correct the position of the device in the same way as            Anchor0 is used. It should be noted that the “North” is            orthogonal to gravity. The device performs calibration for            the up/down direction as it has the IMU necessary for it to            measure and react to gravity. Calibration is performed on a            continual basis. The rate at which calibration is updated            depends on the device.    -   9. The user is instructed to walk to Anchor2. The same process        used to define the label of Anchor1 is performed for Anchor2.        The only difference is that this time the user may walk back and        forth between Anchor2 and either Anchor1 or Anchor0 or both. The        values of the label are calculated in the WCS at each visit to        Anchor2. The same running average algorithm may be used on a per        coordinate basis of the label, i.e., x, y and z.

The WCS synchronization procedure is now complete. LPS system trainingand the building of the network of relocation anchor points (RAPs)commence.

Device Drift Correction Via Optically Enhanced Anchor Points

To measure the location of one WAP relative to another quickly andaccurately, while simultaneously calculating a measurement error bound,typically requires that device drift be corrected. In this method we usean optical identification system to locate the WAP and an anchor pointto provide a local coordinate system around/near the WAP. The act oflocating of the physical WAP ensures that the measurement is taken fromthe same location each time, while the coordinate system provided by theanchor point ensures we can correct for device drift. The device islocatable without drift inside the local coordinate system. There aretwo logically equivalent places to start the measurement from: theorigin of the anchor point and the location of the WAP. For consistencywith anchor points built later in the process, we choose measuring fromthe origin of the anchor point's coordinate system. The pair of theanchor point and optically detected location can be checked forself-consistency before each measurement is taken. Taken together, theabove implies that it is perfectly valid to say that the device takesthe measurement from the same location each time, even though the usermay be standing in a different position each time the measurement istaken.

To measure device drift is then simply the act of measuring theperceived drift of the anchor point when the device returns to the localcoordinate of the anchor point.

Validation of Optically Enhanced Anchor Point

We start with the device close enough to a WAP to make the measurementWAP.p at time T1. Repeated measurements are taken to ensure the locationis accurately known. As soon as the measurement is considered stable,the device creates Anchor and measures Anchor.p. Both WAP.p and Anchor.pare in the same device coordinate system at time T1. Neither one ofthese numbers is an invariant unto itself. If the device were to moveaway and come back, the device will have drifted, and hence the measuredvalues of Anchor.p and WAP.p will be different. For robustness we makeno assumption about the amount of drift in either position ororientation. Hence, at time T2 the device's coordinate system is bestconsidered as independent of the one at time T1. The difference betweenAnchor.p and WAP.p, which we call Delta, is an invariant, however, asthose two numbers can be measured in a sufficiently short time such thatdrift will not accumulate in between the measurements. A simplecomparison for Delta between two successive samples is Peltal. Themagnitudes of Delta from time T1 and T2 are directly comparable withouthaving to find a coordinate transform between the two.

Now assume that the device has made the measurements at times T1 and T2and the Deltas have been calculated. Since we required repeatedmeasurements to validate WAP.p was accurately measured in the devicecoordinate system we can use |Delta| to indicate if the anchor point hasmoved to a different location at time T2. Hence, if |Delta| at T2 isdifferent than at T1, the anchor point is not in the same place. Twothings can be done: the user can look around in attempt to get theanchor point to pop back into position or the anchor point can berebuilt. In the second case it causes a restart for the measurementbeing taken. If |Delta| has not changed, then we can inspect theposition WAP.p in the anchor point's coordinate to see if it haschanged. It is possible, however unlikely, that the location of the WAPwas measured to be equidistant to the anchor point but not in the samerelative location. Likewise, ignoring the check on |Delta| incurs theslight risk that the anchor point and device both drifted in such a waythat the position WAP.p in the local coordinate of the anchor remainsthe same even though the anchor point has shifted.

WAP1 Location Measurement

Now let us examine the process of measuring the distance from Anchor0 toAnchor1. During this process every measurement is validated as above.This ensures that at each measurement step we are in fact measuring fromthe correct physical location. Importantly, this reduces the number ofiterations required for convergence to the average, and hopefully true,value of the measurement. Each iteration corresponds to the end userwalking between WAP0 and WAP1; hence minimizing the number of trips isan important goal.

FIG. 16 shows example drift that a device experiences in movement fromWAP0 to WAP1. In FIG. 16, γ and β are drifts that the device experienceswhen it moves between WAP0 and WAP1. Since we locate the device comparedto a physical point at the extremes of each measurement, we do not haveappreciable measurement error of the location of the start and end ofthe path. Hence, we can calculate the estimated distance

$r_{est} = {\frac{\overset{\_}{r + \beta} + \overset{\_}{r + \gamma}}{2} = {{r + \frac{\overset{¯}{\beta} + \overset{¯}{\gamma}}{2}} = {r + {\overset{¯}{\beta}.}}}}$

as Here we assume that β=γ, because the path lengths the device movesare approximately equal. Please note that the path length is differentthan the measurement; the user walks a path to take the measurement. Thebar represents average value. We also assume that β+γ≠β+γ. In fact, itis quite likely because the two variables are not likely to beindependent. We do know however that β+γ≤β+γ.

The next goal is to find an upper bound on that is directly measurable.The value of β+γ is trivial to measure, but the above inequality showsits lack of utility in achieving a measurable upper bound on β.

FIG. 17 shows an example of accumulated drift. In FIG. 17 we assume thatthe device starts at Position 0 with no drift.

The device moves to Position 1 and experiences a drift of β. The devicethen further moves to Position 2 and accumulates a total drift of a. Fora quality device the averages of the two are most likely 0, i.e., β=α=0.However, if the device does not perform any active drift correction thatbiases the results based on position we can safely assume that /β≤|α|.We do expect that all quality devices will perform drift correction,just not in a way that biases a specific location.

FIG. 18 shows a result of redrawing the FIGS. 16 and 17, with Position 0& 2 at the location of WAP0 and Position 1 being the location of WAP1.

Our device drift correction method enables us to calculate a directly.Combine this with the well-known fact that β≤|β| yields the upper boundwe desire, i.e., β≤|β|≤|α|.

From a training perspective the user takes enough measurements tostabilize the estimate of β. To measure the error bound may require moremeasurements before the estimate of |α| stabilizes.

WAP2 Location Measurement

Next, we consider locating WAP2 in the WCS. This WAP performs twoimportant functions. First, it allows for a direct measure of thestability and robustness of the WCS as defined by the first two. This isneeded so that the network of relocation anchor points can be accuratelyplaced. Secondly, after training, this WAP will provide another physicallocation with a known WCS position. Hence, when the device is near thisWAP, it will be able to calculate its WCS position with tremendousaccuracy. Having three such points available as opposed to two makes theprocess of creating relocation anchor points take less time.

The WCS position of WAP2 is Anchor2.1. As we saw above, there is norestriction to the values of the three coordinates. Hence, simplemagnitude distance measurements will not suffice to determine itslocation in the WCS. The same mathematics hold in calculating theposition of WAP2 as in WAP1. The only difference is that we define β asa vector. The relationship |β|≤|α| still holds and still provides abound to the measurement error.

Calculating the WCS “North” Vector

The WCS “North” vector was defined in the WCS synchronization process asthe vector from an anchor point located near WAP0 to an anchor pointlocated near WAP1. The behavior of anchor points is such that every timean anchor point is relocated in the point cloud map, its position andorientation slightly change. An anchor point defines a local coordinatesystem. One can set the orientation of that coordinate system so that itpoints in the direction of the “North” vector. FIG. 19 illustrates theanchor point position and orientation errors.

The device is also subject to drift and may relocate itself at any timefor any reason. Hence, the error bound around the ideal location of theanchor point is the sum of multiple errors. These errors include:

-   -   Location error of fitting anchor point into point cloud map.    -   Relative drift error, if it exists, of the device compared to        the anchor point. Device drift itself is not a problem because        the drift cancels out in the calculation of the direction        vector.    -   Anchor position estimate error due to device relocation.

At anchor point creation time the forward vector of the anchor point'slocal coordinate system is set to the ideal “North” vector. When theanchor point is placed inside the point cloud map, the anchor pointorientation is subject to a small amount of error. This causes theforward vector of the placed anchor point to be off by an angle of θcompared to the ideal. Even with all of the various sources ofpositional error it has been found experimentally that, when the anchorpoints are sufficiently far apart, the calculated north vector is closerto true than the forward vector of the placed anchor point. Thecalculation of the “North” vector is simply the subtraction of the headpoint (e.g., WAP1) minus the tail point (e.g., WAP0).

If two devices have a good estimate of the “North” vector, then theholograms displayed in those devices will appear in the same location.Experimentally we have found that holograms spread at tens of meters arenot subject to noticeable positional error due to the lever arm effectof mismatched north vectors. This is due to the inability of two usersto share a precise measurement. Obviously, if one user could lookthrough two devices at the same time, smaller errors would becomeperceptible. This is exactly the problem that occurs, however, when oneof the underlying anchor points defining the “North” vector goes bad. Anew pair can be immediately chosen, but the new estimate will beslightly off from the old and a slight pop in positions of hologramsoccurs. This is unacceptable. For this reason, we employ a simpleaveraging technique.

One could average either the forward of vectors of the placed anchorpoints or the calculated north vectors in order to obtain an estimate ofthe true north vector. The forward vectors would simply require moresamples than the calculated north vectors because the variance of thedirectional error of the forward vector has been observed to be higher.Therefore, we use the average of the calculated north vectors over acollection of pairs of anchor points. The calculation of this average isquite simple to perform.

-   -   1. Calculate the angle φ of the direction vector calculated        between the WCS position values of two anchor points. This        calculation is performed on a horizontal plane. Height values of        the two anchor points are ignored. In normal operation the WCS        values of an anchor point don't change; hence, φ can be        calculated and stored for repeated use. Φ is the angle between        the calculated direction vector and the “North” vector (which is        also a direction vector).    -   2. Get the position of the anchor points in the local coordinate        system of the device and create the direction vector in local        coordinates to the device. The device natively supports giving        the position of the anchor points in its local coordinate        system.    -   3. Rotate the direction vector in local coordinates of device by        the angle cp. This yields a sample calculated north vector.    -   4. Average over all sample calculated north vectors to create        the estimate of the “North” vector.

Utilizing this averaging technique implies that when an anchor pointgoes bad and any sample associated with it is removed from the averagingcalculation, the resulting new average value will not noticeably changefrom the previous “North” vector estimate.

Calculating Device WCS Position

One of the goals of the WCS is for devices to be drift free whenmeasured in WCS. We can do nothing about a device's local coordinatesystem drifting. Another goal is that smooth movement of the deviceresults in smooth changes in the measured WCS position of that device.To eliminate drift, we use an anchor point that is close to the deviceto perform the transform from the local coordinate of the device to thatof the WCS. To eliminate pops or ensure smooth measured WCS we take anaverage of the measured WCS from multiple nearby anchor points. Theestimated “North” vector from above along with the device's view ofgravity completes the information required to perform the transforms.

By using a good quality estimate of the “North” vector, we can use anarbitrary number of anchor points in the average, up to and includingall of them. In practice, it is best to use a few close by anchor pointsthat have the lowest measured I al as defined earlier as those anchorpoints will have the best position estimates.

Device Capture into WCS

We describe how a device determines its WCS position on startup or afterthe WCS has been lost. We call this process device capture. We assumethat a collection of relocation anchor points has already been created.Capture is complete when the device has created a “North” vector andvalidated a minimum set of anchor points.

On startup, the system downloads all the anchor points for the device.Once they have been downloaded, they can be visualized by the user withthe help of an attached hologram. Using the holograms as a guide, thesystem guides the user to walk around. The user is instructed to visitthe holograms in order to collect mapping data to help accurately placethe holograms. The process is complete when the anchor points have beendetermined to be in the correct place. This determination is madepossible by the fact that each of the anchor points is labeled with itsWCS coordinate value. At the start of the process, the “North” vectorhas not been estimated. Therefore, in order to determine if an anchorpoint is valid, we check consistency with neighboring anchor points. Thelocal coordinate system of every anchor point is an estimate of the WCS.Given any anchor point, Anchor, we check the location of nearby anchorpoints in Anchor's local coordinate system. If the labeled WCS value ofa nearby anchor point closely matches the WCS estimate in Anchor's localcoordinate system, we say that those two anchor points are consistentwith each other. When a sufficiently large network of self-consistentanchor points has been located, the system generates an estimate of the“North” vector and the device is considered captured.

The density of anchor points determines the ability of the system todetect bad anchor points. If three anchor points are within range ofeach of other to make accurate estimates, then one bad anchor point isdetectable. In general, if N anchor points are within range of eachother, then N−2 anchor points can be determined to be bad. This is basedon the likelihood of a subset of anchor points shifting together in thesame way being unlikely. The caveat for this statement is that theanchor points should be created far enough away from each other suchthat the device does not optimize and combine anchor points to savedevice system resources.

It has been noticed experimentally that all anchor points can shifttogether when the spatial understanding map is corrupted. In this caseit is unlikely that the holograms will appear in a reasonable location.If the LPS system is trained, it can be used to immediately detect thiserror. If LPS is not available, feedback from the user could be used toinform the system that all anchor points have shifted. The hologramsused to indicate the anchor points could be labeled with meaningfulinformation to help the user determine that they are in the wrongposition.

LPS System Training

There are two distinct things being trained in the LPS system:adjacencies and centroids. Because the antenna coordinate system is notsmooth, adjacencies cannot be inferred from the numerical values of theantenna coordinate. On the other hand, the only thing required to trainadjacencies is a minimum time between adjacent locator packets. If thisminimum is met, then we assume that two antenna coordinates areadjacent. Adjacency training is therefore independent of centroidtraining and may commence as soon as the WAPs are installed. Centroidtraining on the other hand requires the WCS. The reason for this issimple: the Cartesian centroid of an antenna coordinate is measured inthe WCS. A device may train an antenna coordinate's centroid if thefollowing two conditions hold.

-   -   WCS “North” vector is defined.    -   Device is within a specified distance of a good quality anchor        point. The specified distance is device dependent, but 3 meters        is a good example.

The above begs the question of the definition of a good quality anchorpoint.

-   -   Anchor points are labeled with their WCS coordinate.    -   The WCS value of a nearby device as measured by the anchor point        closely matches the WCS of the device as measured by LPS, i.e.,        centroid at antenna coordinate of device.    -   The relative positions of adjacent anchor points are within a        defined error bound.

The WCS label on each anchor point permits the measure of this error.

Anchor Point Creation

We now fall back and describe how the anchor points are created in thefirst place. As stated previously, goals of the WCS are to remove devicedrift and ensure smooth changes in measured WCS value with smoothchanges in device position. We also have the goal to reduce positionalerror as compared to the ideal as much as possible. As seen above, thedevice plays a major role in the accuracy of the fundamentalmeasurements. We cannot eliminate positional error as provided by theWCS system. We can however calculate the WCS position of anchor pointsin such a way as to manage the error in the least harmful way possible.

The easiest way to see this is by example. Say we want to create a lineof anchor points between WAP0 and WAP1. We could use WAP0 as thereference and work our way out to WAP1 creating anchor points along theway. The issue here is that this might cause the last anchor point tohave the largest error in WCS position. WAP1's WCS value is fixed afterall. It is this type of bunched up error we view as the most harmful.Positional error should be distributed as evenly as possible so there isless of a chance of a user ever noticing it. We would not for instancewant the shape of a player's avatar to change simply because they are ina different position in the WCS.

Another issue is that the layout of the environment and the placement ofthe WAPs are unique for each installation. Therefore, the algorithm forcreating anchor points preferably allows an anchor point to be createdat any point within the playable area and the anchor points should beable to be created in any order without adversely affecting the qualityof the WCS position.

There exists a small paradox in the process of creating anchor pointsillustrated by the following two conflicting facts.

-   -   The exact position in WCS of an anchor point is trained over        time, becoming fixed only at the end of the training process.    -   Defining the exact position in WCS of an anchor point before        construction is desirable because it simplifies the process of        synchronizing multiple different versions of the WCS together as        well as synchronizing the devices with the WCS manager. The WCS        manager is the piece of logic that determines the locations of        the anchor points in the first place. This logic may be located        on the game console.

Luckily there is a simple solution to this dilemma. The WCS managersimply needs to tell the device where it would like an anchor point togo in the WCS. The device then places the anchor point in that estimatedlocation. Since the WCS is not fully trained at this point, the deviceis still suffering from drift, and hence the WCS location is anestimate. The device then takes repeated measurements of the anchorpoint's position in WCS much in the same way it did for the opticallyenhanced anchor points. When both the observed WCS position and alphavalues stabilize, the anchor point is located within the WCS. Because ofdevice drift, the actual value will be close to but not exactly the sameas the desired value as given by the WCS manager. To rectify this, wesimply use the anchor point for what it is was designed for in the firstplace. We attach a game object tied to the anchor point at the preciseWCS position that the WCS manager wanted the anchor located in the firstplace. The orientation of this game object is set to face in thedirection of the “North” vector. Typically, an anchored object cannotchange orientation but because this object is childed to the anchorpoint, its orientation is not fixed and the orientation may becalibrated if so desired.

Before discussing the process of building further anchor points, the“North” vector must be discussed. During the creation of these newanchor points we have three trained WCS locations. This is not ideal forthe averaging method described before. Since the user is activelytraining the WCS, it is acceptable if loss of tracking of the “North”vector results in a temporary delay of training until tracking isreturned. With the three optically enhanced anchor points we have threedirection vectors that can be averaged together in the manner previouslydescribed. The difference here is that if one of the optically enhancedanchor points is determined to go bad, that anchor point must typicallybe relocated or rebuilt. Worst case the entire process might have tostart from scratch. This unlikely event could happen if the environmentaround one of the WAPs changes significantly.

When initially placing an anchor point, the location in the device'slocal coordinate is calculated by taking the weighted average of theestimated location using each of the three optically enhanced anchorpoints as the origin. The weight is inversely proportional to thedistance from the desired location to the respective optically enhanceanchor point. After creation, the anchor point is physically fixed inlocation so now the process changes to measuring the anchor point'slocation in the WCS. This is performed repeatedly until the WCS locationand alpha value both stabilize. The user is instructed to walk throughthe playable area so that the user passes by the optically enhancedanchor points regularly in order to validate that they have not gonebad. By doing loops such as this, multiple measurements for each anchorpoint are averaged together until the WCS and alpha value arestabilized. Each measurement is similarly the weighted average of theWCS position of the anchor point by using each of the three opticallyenhanced anchor points as the origin. The weight is as previouslydefined. Adjacent anchor points are compared to for self-consistencysince neither the LPS nor optical backup is available at the anchorpoint to check if it has shifted. After the anchor point has stabilized,the child game object is added as described above.

In this way the error in the WCS is spread evenly, and the boundary caseof having error bunched up next to an optically enhanced anchor point isavoided.

Rebuilding an Anchor Point

During normal operation of the game system, an anchor point may go badand need to be rebuilt. It is not acceptable to require user assistancein order to accomplish this; the device must preferably perform theoperation in the background. The loss of a single anchor point is easilyhandled because the density of nearby anchor points is sufficientlyhigh.

-   -   1. Choose the WCS position to place the anchor point.    -   2. Nearby anchor points make an estimate of the WCS location in        the device's local coordinate system.    -   3. The weighted average of the estimate is used to create an        anchor point at the desired location. The weights are once again        the distance from the new anchor point to the anchor used as the        origin.    -   4. Assign new alpha value to the newly created anchor point. The        derivation of the new alpha value is described below.

Over time as anchor points go bad and get recreated, the alpha valueswill tend to increase. The WCS manager may track the alpha values, andwhen a threshold is exceeded, user intervention may be requested torebuild anchor points using the same mechanism to build them in thefirst place. In this way the alpha value would be reset to a smallestvalue possible.

Calculating Alpha Value of Automatically Rebuilt Anchor Point

The alpha value for the newly rebuilt anchor point needs to account forboth the translation errors of the existing points as well as the leverarm effects caused by estimating the location of the new anchor in eachof the existing anchor points' version of the WCS, i.e., “North” vectorplus anchor point as origin. The measure of translation error is thealpha value for the i'th anchor point is |α_(i)|, as calculated above.The relationship is shown in FIG. 20.

In FIG. 20, “Position Error Bound” is |α_(i)|.

The alpha value of the new anchor corresponds to the error bound of itslocation in the WCS. The part due to translation effects is given by

$\alpha_{new} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{\overset{\_}{\alpha_{\iota}}.}}}$

Let |ε_(i)| represent the magnitude of error caused by lever arm effectat the i'th existing anchor point. This is illustrated in the FIG. 21.

We know that the average of ε_(i) over all existing anchor points is 0because that is how the actual location of the anchor point wascalculated. To account for the fact that the new anchor point cannot beas accurately located using existing imprecise anchor points we use theheuristic of

$\frac{1}{N}{\sum\limits_{i = 1}^{N}{ɛ_{i}}}$

to add to the alpha value of the new anchor point. Doing this gives amethod for the WCS manager to decide if user intervention should beapplied to rebuild the anchor point more accurately. The final alphavalue for the new anchor point is therefore,

$\alpha_{new} = {{\frac{1}{N}{\sum\limits_{i = 1}^{N}\overset{\_}{\alpha_{\iota}}}} + {\frac{1}{N}{\sum\limits_{i = 1}^{N}{{ɛ_{i}}.}}}}$

Part B: Use of the WCS for Perception Management

The WCS supports many important features that enable it to be used tomanage the user's visual perception of holograms within the environment.These include:

-   -   1. A common language for devices of different types to        communicate in.    -   2. Placement of holograms independent of environment.    -   3. Simplified creation and maintenance of virtual models of the        environment. Works across device families.    -   4. Centrally controlled occlusion management.    -   5. Accurate perception of hologram location.

WCS—The Common Language

Having a coordinate system that is understood by all participatingdevices/subsystems means that the devices share a common language tocommunicate about game objects such as holograms and physical players.This reduces the amount of information that must be shared at run timefor a game to be supported. FIG. 22 illustrates an example of thisarrangement.

During runtime the information exchanged is WCS. This means that eventhough a multiplayer multilocation AR game is being supported, from aninformation exchange standpoint it is not much different than a legacyconsole multiplayer game. Keeping the exchange of detailed anchor pointinformation local and primarily at setup time is a big innovation forReavire AR games.

Hologram Placement

Holograms are placed in a game in the WCS directly. The implications ofthis are very important. There is no need to create an anchor point tostabilize the hologram because the WCS is itself actively stabilized.FIG. 23 illustrates and example of this arrangement.

The arrangement of FIG. 23 allows game designers to focus on making fungames and not have to worry about stabilizing holograms and managinganchor points. This makes AR games on the Reavire platform easier todevelop than a game directly built on AR glasses. In addition, at runtime, Reavire AR games will not have to constantly share anchor pointsbetween the local players because they have already been shared pregameand persist throughout the entirety of game play. This is particularlyimportant for high action multiplayer games because the number ofholograms is expected to be quite large. In legacy AR games, each newcreated anchor point needs to be shared with other players leading topossible congestion on the local network, negatively impacting gameplay.

Another important point is that a centralized game engine can creategame objects at will without worry about how they will be placed in aspace about which it has very limited knowledge. The only thing thecentral server needs to know is where holograms can be placed, i.e., asimplified understanding of the playable area.

Additionally, holograms can be placed at great distances from theplayable area itself. The WCS is stable enough to show hologrampositions accurately at many 10's of meters.

Virtual Models

The creation of virtual models always starts with optical techniques.The device scans the environment and creates a surface mesh of theenvironment. Most devices include functions or applications to derivehigher order geometry from these meshes, such as planes. This in turn isused to detect walls, floor, ceiling, tabletops, etc. The addition ofthe WCS permits these higher order objects to be efficiently persistedand managed.

In managing an object, e.g., a wall, a user may adjust the controlpoints on a wall object to more closely match the corresponding physicalwall. The wall object and its control points are in the WCS so now anyother device can load that wall object and have it appear in the correctspot without having to go through the process of identifying andconstructing the wall from scratch. In this way the entire playable canbe mapped into a virtual model that is defined in the WCS. When a deviceconnects, this virtual model is downloaded to the device and entirevirtual model is available to the device without requiring the player tomap the playable area before the game ensues. This is in stark contrastto existing AR built on glasses such as the HoloLens, which require theuser to scan the area at the beginning of game play. The stability andaccuracy of the WCS permit the entire model to precisely match actualgeometry even as the player moves around the playable area.

Occlusion

Occlusion is a complicated subject because different devices occludeholograms in different ways. Some devices make occlusion masks, and thenuse those masks to perform occlusion during the course of the game. Somedevices actively occlude all the time, constantly mapping the visiblesurfaces to create occlusion masks. For devices that create staticocclusion masks there is a wide variety of methods in treating the maskafter it has been created. The fundamental idea is that the mask isintended to be coherent with the physical objects it is performing theocclusion for. Listed below are three possible methods of managing thepremade masks. We assume that the mask is implemented as invisible mesh.

-   -   1. The mesh is fit using a best-fit technique similar to how        anchor points are fit into the point cloud map.    -   2. Mask is stored as one big mesh and attached to an anchor        point.    -   3. Mask is broken into pieces and each piece is attached to its        anchor point.

Furthermore, the method that the device uses to translate a holograminto its local coordinate to view is a design choice when the hologramsare anchored. This affects the position of the viewed holograms. We haveno control over either of the above design choices the device made. Thestatic mask method is common because it cuts down the device'scomputational load in game, saving battery life and improving overallperformance.

The problem is that Reavire games are both multi-device and multi-mode.Physical players may use different devices and non-local players viewthe local terrain in virtual reality mode.

The display behavior of unanchored objects is well defined by devices.The object is defined and shown in the coordinate system of the device.If the device drifts, the object drifts with it. But the relationshipbetween unanchored holograms never changes.

For all these reasons, in the Reavire AR game system, occlusion masksare defined by the simplified virtual model of the playable area. Thisensures that all devices behave in the same way and that the game isfair between both physical and virtual players.

Perception

The perceived location of any object is aided by many clues: occlusion,lighting, size, etc. The challenge in AR is to provide enough clues suchthat the player can tell where the hologram is supposed to be. It hasbeen noticed experimentally that without help it is virtually impossibleto know where a hologram is when it is past a certain distance from theuser. FIG. 24 shows a user looking at a sphere without any contextualclues as to the location.

The user cannot tell if he or she is looking at small sphere close tohim or larger sphere farther away because without context they lookidentical.

Size is an important clue in judging the distance of an object, so ifthe viewer knows the size of the sphere beforehand, a guess can be made.Experiments have shown that size alone is not enough of a clue todetermine the distance.

A very important clue in judging object position is lighting. Physicalobjects interact with light in the environment and our eyes have becomeattuned to this, giving us many valuable clues as to the objectlocation. Although it is theoretically possible to add lighting effectsto holograms in an attempt to provide these clues, it is a verycomputationally expensive operation. For this reason, a minimum set ofclues is desired to aid the player in determining hologram location.

The simplified virtual model provides an accurate understanding of thelocation of walls, floors, and ceilings. For the purposes of specialclues, the virtual model is not visualized. It is used for hologramocclusion. Hologram occlusion provided by the virtual model is the firstmajor clue of hologram location.

The next step is to draw lines at the intersection of the walls, floors,and ceilings. This provides a contextual bridge between the real worldand the holograms. Grid lines are then added at regular intervals on asubset of these surfaces. This provides distance contextualization. Thegrid lines tend to work better when they are roughly orthogonal to theuser's gaze. Finally, the hologram's location is highlighted on the gridlines either with a grid of its own or a shadowing effect. With this setof clues, it has been experimentally verified that a user can quicklyand accurately learn to locate the position of the hologram.

The visuals do not need to be vivid; when they are faint they performthe function well and do not interfere with game play.

Having described certain embodiments, numerous alternative embodimentsor variations can be made. For example, the WCS as described herein isalso useful for many applications beyond AR. Applications such asrobotics and drones that rely on SLAM in a similar manner to AR systemswould realize analogous benefits. A wide variety of Internet of Things(IoT) applications could also benefit from the WCS providing stable,reliable location services.

Further, although features have been shown and described with referenceto particular embodiments hereof, such features may be included andhereby are included in any of the disclosed embodiments and theirvariants. Thus, it is understood that features disclosed in connectionwith any embodiment are included in any other embodiment.

Further still, the improvement or portions thereof may be embodied as acomputer program product including one or more non-transient,computer-readable storage media, such as a magnetic disk, magnetic tape,compact disk, DVD, optical disk, flash drive, solid state drive, SD(Secure Digital) chip or device, Application Specific Integrated Circuit(ASIC), Field Programmable Gate Array (FPGA), and/or the like (shown byway of example as media 850 and 1150 in FIGS. 8 and 15). Any number ofcomputer-readable media may be used. The media may be encoded withinstructions which, when executed on one or more computers or otherprocessors, perform the process or processes described herein. Suchmedia may be considered articles of manufacture or machines, and may betransportable from one machine to another.

As used throughout this document, the words “comprising,” “including,”“containing,” and “having” are intended to set forth certain items,steps, elements, or aspects of something in an open-ended fashion. Also,as used herein and unless a specific statement is made to the contrary,the word “set” means one or more of something. This is the caseregardless of whether the phrase “set of” is followed by a singular orplural object and regardless of whether it is conjugated with a singularor plural verb. Further, although ordinal expressions, such as “first,”“second,” “third,” and so on, may be used as adjectives herein, suchordinal expressions are used for identification purposes and, unlessspecifically indicated, are not intended to imply any ordering orsequence. Thus, for example, a “second” event may take place before orafter a “first event,” or even if no first event ever occurs. Inaddition, an identification herein of a particular element, feature, oract as being a “first” such element, feature, or act should not beconstrued as requiring that there must also be a “second” or other suchelement, feature or act. Rather, the “first” item may be the only one.Although certain embodiments are disclosed herein, it is understood thatthese are provided by way of example only and should not be construed aslimiting.

Those skilled in the art will therefore understand that various changesin form and detail may be made to the embodiments disclosed hereinwithout departing from the scope of the following claims.

What is claimed is:
 1. A method, comprising: constructing a WCS (worldcoordinate system) as a three-dimensional, Cartesian coordinate systembased at least in part on (i) a plurality of anchor points detected in aphysical space around a device and (ii) a secondary source ofinformation that assists in localizing objects in the physical space;and providing the WCS to the device as a coordinate system in which toplace holograms.
 2. The method of claim 1, wherein the secondary sourceincludes a plurality of VISOs (visually-identifiable stationary objects)disposed in respective locations of the physical space.
 3. The method ofclaim 1, wherein the secondary source includes an LPS (local positioningsystem) that provides locations of devices in the physical space as anindependent observer.
 4. The method of claim 2, wherein the plurality ofVISOs includes a plurality of WAPs (wireless access points), each of theplurality of WAPs configured to communicate wirelessly with the deviceand having a respective registration mark that is optically detectableby the device.
 5. The method of claim 2, wherein constructing the WCSincludes: defining a first anchor point of the plurality of anchorpoints, the first anchor point associated with a first VISO of theplurality of VISOs; assigning an origin of the WCS based on a locationof the first anchor point; and assigning a first axis of the WCS basedon a direction of gravity measured by the device.
 6. The method of claim5, wherein constructing the WCS further includes: defining a secondanchor point of the plurality of anchor points, the second anchor pointassociated with a second VISO of the plurality of VISOs; assigning asecond axis of the WCS based at least in part on the first anchor pointand the second anchor point; and assigning a third axis of the WCSorthogonal to both the first axis and the second axis.
 7. The method ofclaim 6, further comprising selecting the first VISO and the second VISOas respective VISOs of the plurality of VISOs that are located farthestapart within the physical space.
 8. The method of claim 6, whereinconstructing the WCS further includes: defining a third anchor point ofthe plurality of anchor points, the third anchor point associated with athird VISO of the plurality of VISOs; and verifying stability of thefirst anchor point and the second anchor point using the third anchorpoint.
 9. The method of claim 5, wherein assigning the origin of the WCSas the location of the first anchor point includes repeatedly obtainingcoordinates of the first VISO in a coordinate system of the first anchorpoint and averaging the coordinates.
 10. The method of claim 5, furthercomprising; identifying an additional anchor point within the physicalspace; and rendering a location of the additional anchor point as a setof coordinates in the WCS.
 11. The method of claim 2, wherein theplurality of anchor points includes a first anchor point and a secondanchor point, and wherein constructing the WCS includes identifying aNorth vector of the WCS based at least in part on (i) a position of thefirst anchor point in a coordinate system of the second anchor point and(ii) a position of the second anchor point in a coordinate system of thefirst anchor point.
 12. The method of claim 11, further comprisingupdating the North vector of the WCS in real time based on currentposition estimates of the first anchor point and/or the second anchorpoint.
 13. The method of claim 11, further comprising: detecting that ananchor point of the plurality of anchor points has moved from aninitially established location in the WCS; and deleting the anchor pointso that the deleted anchor point does not contribute to identifying theNorth vector of the WCS.
 14. The method of claim 13, wherein detectingthat the anchor point has moved includes: identifying an error bound fora location of the anchor point based on at least one of (i) inputreceived from an LPS (local positioning system) and (ii) a VISOassociated with the anchor point; and determining that the anchor pointis outside the error bound.
 15. The method of claim 1, furthercomprising: storing the WCS on a server; and providing the WCS to asecond device to enable the second device to render holograms referencedto the WCS.
 16. The method of claim 1, further comprising, upon startingthe device: detecting an anchor point within a field of view of thedevice; orienting the WCS based on pre-established coordinates of thedetected anchor point in the WCS; and placing, by the device, a set ofholograms referenced to the WCS.
 17. A computerized system, comprisingcontrol circuitry that includes a set of processors and memory, thecontrol circuitry configured to perform activities that include:constructing a WCS (world coordinate system) as a three-dimensional,Cartesian coordinate system based at least in part on (i) multipleanchor points detected in a physical space around a set of devices and(ii) a secondary source of information that assists in localizingobjects in the physical space; and providing the WCS to at least one ofthe set of devices as a coordinate system in which to place holograms.18. The computerized system of claim 17, wherein the secondary sourceincludes at least one of (i) a plurality of VISOs (visually-identifiablestationary objects) disposed in respective locations of the physicalspace or (ii) an LPS (local positioning system) that provides locationsof devices in the physical space as an independent observer.
 19. Thecomputerized system of claim 18, wherein the plurality of VISOs includesa plurality of WAPs (wireless access points), each of the plurality ofWAPs configured to communicate wirelessly with the device and having arespective registration mark that is optically detectable by the device.20. A method of locating objects in three-dimensional space, comprising:constructing a WCS (world coordinate system) based on a plurality ofVISOs (visually-identifiable stationary objects) disposed in respectivelocations of a physical space; and providing the WCS to a device as acoordinate system in which to place holograms.
 21. The method of claim20, further comprising establishing an origin and multiple axes of theWCS based on anchor points associated with the plurality of VISOs.